2024-04-16T06:56:28.201 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T06:56:28.201 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi --owner scheduled_vshankar@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --archive /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944 -- /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/orig.config.yaml 2024-04-16T06:56:28.249 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T06:56:28.336 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944 branch: wip-vshankar-testing-20240412.042930 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: vshankar@redhat.com first_in_suite: false job_id: '7657944' kernel: flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/wip-64502-quincy-3 ' name: vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240412.042930 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-64502-quincy-3 sha1: a2afef7120fdaef83ddff164fdad44bc773a2630 owner: scheduled_vshankar@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4053 sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: wip-64502-quincy-3 suite_path: /home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa suite_relpath: qa suite_repo: https://github.com/vshankar/ceph.git suite_sha1: a2afef7120fdaef83ddff164fdad44bc773a2630 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASqUji7Hg2IWkjy40yk/OzXBEAdIfVccvNsVVQ/E6HtRn4MKTqb+h1uO7U++GCAjvODh2qCgdlaUuqWd9LclDU= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP742jjOnmsSmk4FOLUX8fmmQlWDuINa4UBsV3aBnYtJ2DZ+7N6h0yngVBpyYKn4C8bhJop0/cCuq344DfCVtG0= tasks: - install: branch: wip-64502-quincy-3 exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/vshankar/src/github.com_vshankar_ceph_wip-64502-quincy-3/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_06:50:35 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T06:56:28.336 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa; will attempt to use it 2024-04-16T06:56:28.337 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks 2024-04-16T06:56:28.337 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T06:56:28.338 INFO:teuthology.task.internal:Checking packages... 2024-04-16T06:56:28.397 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '218a691417ef57c5d22a8d5cf3c5f41f7b542838' 2024-04-16T06:56:28.397 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T06:56:28.397 INFO:teuthology.packaging:ref: None 2024-04-16T06:56:28.397 INFO:teuthology.packaging:tag: None 2024-04-16T06:56:28.397 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240412.042930 2024-04-16T06:56:28.397 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T06:56:28.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240412.042930 2024-04-16T06:56:28.614 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2884.g218a6914 2024-04-16T06:56:28.615 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T06:56:28.623 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T06:56:28.623 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T06:56:28.679 INFO:teuthology.task.internal:Saving configuration 2024-04-16T06:56:28.697 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T06:56:28.708 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T06:56:28.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 06:51:33.701669', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASqUji7Hg2IWkjy40yk/OzXBEAdIfVccvNsVVQ/E6HtRn4MKTqb+h1uO7U++GCAjvODh2qCgdlaUuqWd9LclDU='} 2024-04-16T06:56:28.754 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 06:51:33.702665', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP742jjOnmsSmk4FOLUX8fmmQlWDuINa4UBsV3aBnYtJ2DZ+7N6h0yngVBpyYKn4C8bhJop0/cCuq344DfCVtG0='} 2024-04-16T06:56:28.755 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T06:56:28.761 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-16T06:56:28.761 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-16T06:56:28.761 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T06:56:28.891 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ce82630d0>, signals=[15]) 2024-04-16T06:56:28.892 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T06:56:28.897 INFO:teuthology.task.internal:Opening connections... 2024-04-16T06:56:28.897 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T06:56:28.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T06:56:29.002 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-04-16T06:56:29.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T06:56:29.086 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T06:56:29.093 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-04-16T06:56:29.110 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-04-16T06:56:29.110 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T06:56:29.167 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-04-16T06:56:29.168 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T06:56:29.168 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T06:56:29.168 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T06:56:29.168 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-04-16T06:56:29.191 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-04-16T06:56:29.208 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-04-16T06:56:29.208 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-04-16T06:56:29.263 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T06:56:29.264 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T06:56:29.265 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-04-16T06:56:29.282 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T06:56:29.292 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T06:56:29.325 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T06:56:29.326 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-04-16T06:56:29.328 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-04-16T06:56:29.343 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T06:56:29.364 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T06:56:29.364 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-04-16T06:56:29.385 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-04-16T06:56:29.417 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T06:56:29.472 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T06:56:29.486 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-16T06:56:29.486 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}}, timeout 300 2024-04-16T06:56:29.486 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T06:56:29.487 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T06:56:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:56:29.502 DEBUG:teuthology.task.kernel:sha1 for host.b is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.502 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T06:56:29.502 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-16T06:56:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:56:29.506 DEBUG:teuthology.task.kernel:sha1 for host.a is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.506 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T06:56:29.506 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-04-16T06:56:29.558 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-361.el9.x86_64 2024-04-16T06:56:29.559 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.559 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T06:56:29.559 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.563 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-04-16T06:56:29.563 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.563 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T06:56:29.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:56:29.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum upgrade -y linux-firmware 2024-04-16T06:56:29.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum upgrade -y linux-firmware 2024-04-16T06:56:31.936 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T06:56:31.957 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-16T06:56:33.437 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-04-16T06:56:33.585 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-04-16T06:56:34.335 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-16T06:56:35.150 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-04-16T06:56:37.585 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-16T06:56:41.585 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-16T06:56:43.327 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-16T06:56:45.506 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-16T06:56:50.503 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-16T06:56:52.953 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T06:56:52.960 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T06:56:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 1 Package 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:56:52.961 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 375 M 2024-04-16T06:56:52.962 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 934 kB/s | 19 MB 00:20 2024-04-16T06:57:00.061 INFO:teuthology.orchestra.run.smithi104.stdout:linux-firmware-20240219-143.el9.noarch.rpm 53 MB/s | 375 MB 00:07 2024-04-16T06:57:00.062 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:57:00.062 INFO:teuthology.orchestra.run.smithi104.stdout:Total 53 MB/s | 375 MB 00:07 2024-04-16T06:57:00.063 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T06:57:00.149 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T06:57:00.149 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T06:57:00.342 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-16T06:57:01.985 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-16T06:57:02.592 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T06:57:02.592 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T06:57:03.045 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-16T06:57:08.287 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-16T06:57:09.076 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T06:57:09.345 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T06:57:09.433 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:10.798 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:10.798 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T06:57:10.853 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:57:10.863 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T06:57:10.864 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T06:57:10.864 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 1 Package 2024-04-16T06:57:10.864 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:57:10.864 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 375 M 2024-04-16T06:57:10.864 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T06:57:11.196 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:11.197 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:57:11.197 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-04-16T06:57:11.197 INFO:teuthology.orchestra.run.smithi104.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T06:57:11.197 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:57:11.197 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T06:57:11.492 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T06:57:11.492 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:57:11.650 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T06:57:11.651 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T06:57:11.653 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi104.front.sepia.ceph.com to complete... 2024-04-16T06:57:17.679 INFO:teuthology.orchestra.run.smithi026.stdout:linux-firmware-20240219-143.el9.noarch.rpm 55 MB/s | 375 MB 00:06 2024-04-16T06:57:17.680 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:57:17.680 INFO:teuthology.orchestra.run.smithi026.stdout:Total 55 MB/s | 375 MB 00:06 2024-04-16T06:57:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T06:57:17.775 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T06:57:17.775 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T06:57:20.331 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T06:57:20.331 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T06:57:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16 06:57:24 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T06:57:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:FINISHED --2024-04-16 06:57:24-- 2024-04-16T06:57:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:Total wall clock time: 12s 2024-04-16T06:57:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:Downloaded: 1 files, 341M in 12s (28.2 MB/s) 2024-04-16T06:57:24.050 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T06:57:24.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T06:57:26.099 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying... ######################################## 2024-04-16T06:57:26.176 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing... ######################################## 2024-04-16T06:57:26.257 INFO:teuthology.orchestra.run.smithi104.stdout:Updating / installing... 2024-04-16T06:57:26.905 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T06:57:27.151 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T06:57:27.248 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T06:57:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T06:57:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:57:29.114 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-04-16T06:57:29.114 INFO:teuthology.orchestra.run.smithi026.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T06:57:29.114 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:57:29.114 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T06:57:29.383 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T06:57:29.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T06:57:29.769 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T06:57:29.769 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T06:57:29.772 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi026.front.sepia.ceph.com to complete... 2024-04-16T06:57:34.202 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T06:57:41.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16 06:57:41 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T06:57:41.881 INFO:teuthology.orchestra.run.smithi026.stderr:FINISHED --2024-04-16 06:57:41-- 2024-04-16T06:57:41.881 INFO:teuthology.orchestra.run.smithi026.stderr:Total wall clock time: 12s 2024-04-16T06:57:41.881 INFO:teuthology.orchestra.run.smithi026.stderr:Downloaded: 1 files, 341M in 12s (28.9 MB/s) 2024-04-16T06:57:41.884 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T06:57:41.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T06:57:44.043 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying... ######################################## 2024-04-16T06:57:44.098 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing... ######################################## 2024-04-16T06:57:44.182 INFO:teuthology.orchestra.run.smithi026.stdout:Updating / installing... 2024-04-16T06:57:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T06:57:52.579 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T06:58:09.448 INFO:teuthology.orchestra.run.smithi026.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T06:58:13.487 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T06:58:13.487 DEBUG:teuthology.orchestra.run.smithi104:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T06:58:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T06:58:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T06:58:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T06:58:13.560 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T06:58:13.561 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T06:58:13.562 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T06:58:13.563 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T06:58:13.564 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T06:58:13.565 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T06:58:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T06:58:13.567 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T06:58:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T06:58:13.569 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T06:58:13.570 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T06:58:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T06:58:13.572 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T06:58:13.573 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T06:58:13.574 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T06:58:13.575 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T06:58:13.576 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T06:58:13.577 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T06:58:13.578 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T06:58:13.579 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T06:58:13.580 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T06:58:13.581 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T06:58:13.582 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T06:58:13.583 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T06:58:13.584 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T06:58:13.585 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T06:58:13.586 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T06:58:13.587 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T06:58:13.588 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T06:58:13.589 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T06:58:13.590 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T06:58:13.591 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T06:58:13.592 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T06:58:13.593 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T06:58:13.594 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T06:58:13.595 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T06:58:13.596 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T06:58:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T06:58:13.598 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T06:58:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T06:58:13.600 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T06:58:13.601 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T06:58:13.602 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T06:58:13.603 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T06:58:13.604 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T06:58:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T06:58:13.606 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T06:58:13.607 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T06:58:13.608 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T06:58:13.609 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T06:58:13.610 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T06:58:13.611 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T06:58:13.612 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T06:58:13.613 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T06:58:13.614 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T06:58:13.615 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T06:58:13.616 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T06:58:13.617 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T06:58:13.618 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T06:58:13.619 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T06:58:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T06:58:13.621 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T06:58:13.622 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T06:58:13.623 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T06:58:13.624 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T06:58:13.625 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T06:58:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T06:58:13.627 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T06:58:13.628 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T06:58:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T06:58:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T06:58:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T06:58:13.632 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T06:58:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T06:58:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T06:58:13.635 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T06:58:13.636 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T06:58:13.637 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T06:58:13.638 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T06:58:13.639 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T06:58:13.640 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T06:58:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T06:58:13.642 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T06:58:13.643 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T06:58:13.644 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T06:58:13.645 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T06:58:13.646 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T06:58:13.647 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T06:58:13.648 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T06:58:13.649 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T06:58:13.650 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T06:58:13.651 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T06:58:13.652 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T06:58:13.653 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T06:58:13.654 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T06:58:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T06:58:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T06:58:13.657 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T06:58:13.658 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T06:58:13.659 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T06:58:13.660 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T06:58:13.661 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T06:58:13.662 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T06:58:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T06:58:13.664 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T06:58:13.665 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T06:58:13.666 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T06:58:13.667 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T06:58:13.668 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T06:58:13.669 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T06:58:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T06:58:13.671 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T06:58:13.672 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T06:58:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T06:58:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T06:58:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T06:58:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T06:58:13.677 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T06:58:13.678 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T06:58:13.679 INFO:teuthology.orchestra.run.smithi104.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T06:58:13.683 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.683 DEBUG:teuthology.orchestra.run.smithi104:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T06:58:13.705 INFO:teuthology.orchestra.run.smithi104.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:fi 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:for file in vmlinuz System.map config; do 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: fi 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:done 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:fi 2024-04-16T06:58:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout:fi 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.707 INFO:teuthology.orchestra.run.smithi104.stdout:fi 2024-04-16T06:58:13.707 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 68.el9 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8264254 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-04-16T06:58:13.786 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T06:58:13.787 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T06:58:13.788 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T06:58:13.788 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:13.788 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T06:58:14.430 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-04-16T06:58:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T06:58:15.797 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-04-16T06:58:15.800 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-04-16T06:58:15.814 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.wYYWXuZwBh 2024-04-16T06:58:15.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wYYWXuZwBh 2024-04-16T06:58:15.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.wYYWXuZwBh 2024-04-16T06:58:16.023 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.wYYWXuZwBh is 6KB 2024-04-16T06:58:16.075 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.wYYWXuZwBh 2024-04-16T06:58:16.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T06:58:16.154 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T06:58:16.154 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T06:58:16.155 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T06:58:16.155 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T06:58:16.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:16.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-04-16T06:58:33.381 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T06:58:33.381 DEBUG:teuthology.orchestra.run.smithi026:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T06:58:33.448 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T06:58:33.449 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T06:58:33.450 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T06:58:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T06:58:33.452 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T06:58:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T06:58:33.454 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T06:58:33.455 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T06:58:33.456 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T06:58:33.457 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T06:58:33.458 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T06:58:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T06:58:33.460 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T06:58:33.461 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T06:58:33.462 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T06:58:33.463 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T06:58:33.464 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T06:58:33.465 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T06:58:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T06:58:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T06:58:33.468 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T06:58:33.469 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T06:58:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T06:58:33.471 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T06:58:33.472 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T06:58:33.473 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T06:58:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T06:58:33.475 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T06:58:33.476 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T06:58:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T06:58:33.478 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T06:58:33.479 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T06:58:33.480 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T06:58:33.481 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T06:58:33.482 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T06:58:33.483 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T06:58:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T06:58:33.485 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T06:58:33.486 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T06:58:33.487 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T06:58:33.488 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T06:58:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T06:58:33.490 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T06:58:33.491 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T06:58:33.492 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T06:58:33.493 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T06:58:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T06:58:33.495 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T06:58:33.496 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T06:58:33.497 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T06:58:33.498 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T06:58:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T06:58:33.500 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T06:58:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T06:58:33.502 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T06:58:33.503 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T06:58:33.504 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T06:58:33.505 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T06:58:33.506 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T06:58:33.507 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T06:58:33.508 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T06:58:33.509 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T06:58:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T06:58:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T06:58:33.512 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T06:58:33.513 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T06:58:33.514 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T06:58:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T06:58:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T06:58:33.517 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T06:58:33.518 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T06:58:33.519 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T06:58:33.520 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T06:58:33.521 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T06:58:33.522 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T06:58:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T06:58:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T06:58:33.525 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T06:58:33.526 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T06:58:33.527 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T06:58:33.528 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T06:58:33.529 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T06:58:33.530 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T06:58:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T06:58:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T06:58:33.533 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T06:58:33.534 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T06:58:33.535 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T06:58:33.536 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T06:58:33.537 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T06:58:33.538 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T06:58:33.539 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T06:58:33.540 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T06:58:33.541 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T06:58:33.542 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T06:58:33.543 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T06:58:33.544 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T06:58:33.545 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T06:58:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T06:58:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T06:58:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T06:58:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T06:58:33.550 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T06:58:33.551 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T06:58:33.552 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T06:58:33.553 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T06:58:33.554 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T06:58:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T06:58:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T06:58:33.557 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T06:58:33.558 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T06:58:33.559 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T06:58:33.560 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T06:58:33.561 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T06:58:33.562 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T06:58:33.563 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T06:58:33.564 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T06:58:33.565 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T06:58:33.566 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T06:58:33.567 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T06:58:33.568 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T06:58:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T06:58:33.570 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T06:58:33.571 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T06:58:33.572 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T06:58:33.573 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T06:58:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T06:58:33.575 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T06:58:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T06:58:33.580 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.580 DEBUG:teuthology.orchestra.run.smithi026:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:fi 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:for file in vmlinuz System.map config; do 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: fi 2024-04-16T06:58:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:done 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:fi 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:fi 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:fi 2024-04-16T06:58:33.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-04-16T06:58:33.690 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T06:58:33.691 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-04-16T06:58:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T06:58:33.692 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T06:58:33.692 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T06:58:33.692 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:33.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T06:58:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-04-16T06:58:35.857 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T06:58:35.876 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-04-16T06:58:35.879 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-04-16T06:58:35.933 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.Jxbm8s9zf1 2024-04-16T06:58:35.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Jxbm8s9zf1 2024-04-16T06:58:36.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.Jxbm8s9zf1 2024-04-16T06:58:36.149 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.Jxbm8s9zf1 is 6KB 2024-04-16T06:58:36.197 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.Jxbm8s9zf1 2024-04-16T06:58:36.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T06:58:36.280 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T06:58:36.280 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T06:58:36.280 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T06:58:36.280 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T06:58:36.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T06:58:36.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-04-16T06:58:46.310 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-16T06:58:46.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T06:58:46.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T06:59:04.863 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-16T06:59:06.473 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-16T06:59:06.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T06:59:06.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T06:59:13.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T06:59:13.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T06:59:16.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-16T06:59:28.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T06:59:28.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:00:00.794 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-16T07:00:01.388 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T07:00:01.388 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T07:00:01.388 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-16T07:00:01.426 INFO:teuthology.orchestra.run.smithi104.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T07:00:01.426 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T07:00:01.426 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:00:01.426 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:00:02.427 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-16T07:00:02.427 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:00:02.555 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-04-16T07:00:02.604 DEBUG:teuthology.parallel:result is None 2024-04-16T07:00:06.476 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:00:15.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T07:00:15.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:00:16.092 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-04-16T07:00:16.731 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T07:00:16.731 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T07:00:16.731 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-04-16T07:00:16.773 INFO:teuthology.orchestra.run.smithi026.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T07:00:16.773 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T07:00:16.773 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:00:16.773 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:00:17.774 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-16T07:00:17.774 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:00:17.916 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-04-16T07:00:17.967 DEBUG:teuthology.parallel:result is None 2024-04-16T07:00:17.968 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:00:17.976 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:00:17.976 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:00:17.980 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:00:18.037 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:00:18.043 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:00:18.052 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:00:18.053 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:00:18.058 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:00:18.128 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:00:18.134 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:00:18.135 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:00:18.139 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:00:18.204 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:00:18.237 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:00:18.242 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:00:18.285 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:00:18.289 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:00:18.296 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:00:18.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:00:18.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:00:18.394 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:00:18.426 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:00:18.427 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:00:18.431 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:00:18.484 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:00:18.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:00:18.675 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:00:18.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:00:18.831 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:00:18.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:00:18.900 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:00:18.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:00:19.113 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:00:19.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:00:19.293 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:00:19.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:00:19.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-04-16T07:00:19.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-04-16T07:00:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:00:19.550 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:00:19.911 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:00:19.919 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:00:19.919 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:00:19.969 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:00:19.982 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:00:19.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-04-16T07:00:20.120 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-04-16T07:00:20.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-04-16T07:00:20.265 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-04-16T07:00:20.271 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:00:20.271 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-04-16T07:00:20.579 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-04-16T07:00:20.579 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-04-16T07:00:20.641 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-04-16T07:00:20.641 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:00:20.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:00:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:00:20.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:00:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:00:20.847 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:00:20.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:00:21.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:00:21.104 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:00:21.145 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T07:00:21.204 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T07:00:24.084 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:00:25.207 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:00:25.242 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:06:55.234 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-04-16T07:06:55.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T07:06:55.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:06:55.386 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-04-16T07:06:55.550 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T07:06:55.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T07:06:55.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:06:55.693 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-16T07:06:55.849 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-16T07:06:55.849 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:06:55.861 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:06:55.861 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:06:55.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:06:55.869 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:06:55.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:06:55.968 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:06:55.970 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:06:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:06:56.028 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:06:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-04-16T07:06:56.127 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-04-16T07:06:56.158 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-04-16T07:06:56.172 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-04-16T07:06:56.209 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:06:56.229 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:06:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:06:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:06:56.394 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.403 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.430 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.441 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:06:56.443 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:06:56.451 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:06:56.451 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'}} 2024-04-16T07:06:56.451 DEBUG:teuthology.task.install:config {'branch': 'wip-64502-quincy-3', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T07:06:56.452 INFO:teuthology.task.install:Using flavor: default 2024-04-16T07:06:56.457 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:06:56.457 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:06:56.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T07:06:56.458 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:06:56.458 INFO:teuthology.packaging:ref: None 2024-04-16T07:06:56.458 INFO:teuthology.packaging:tag: None 2024-04-16T07:06:56.458 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:06:56.458 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:06:56.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T07:06:56.461 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T07:06:56.461 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:06:56.461 INFO:teuthology.packaging:ref: None 2024-04-16T07:06:56.461 INFO:teuthology.packaging:tag: None 2024-04-16T07:06:56.461 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:06:56.461 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:06:56.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T07:06:56.709 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T07:06:56.709 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T07:06:56.715 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T07:06:56.716 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T07:06:56.889 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:06:56.889 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:06:56.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:06:56.895 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:06:56.895 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:06:56.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:06:56.989 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:06:56.989 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:06:56.989 INFO:teuthology.packaging:ref: None 2024-04-16T07:06:56.989 INFO:teuthology.packaging:tag: None 2024-04-16T07:06:56.989 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:06:56.989 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:06:56.990 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:06:57.004 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:06:57.004 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:06:57.004 INFO:teuthology.packaging:ref: None 2024-04-16T07:06:57.004 INFO:teuthology.packaging:tag: None 2024-04-16T07:06:57.004 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:06:57.004 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:06:57.004 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:06:57.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:06:57.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:06:57.262 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:06:57.291 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:06:57.349 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-04-16T07:06:57.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-04-16T07:06:57.382 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-04-16T07:06:57.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-04-16T07:06:57.702 INFO:teuthology.orchestra.run.smithi104.stdout:74 files removed 2024-04-16T07:06:57.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-04-16T07:06:57.766 INFO:teuthology.orchestra.run.smithi026.stdout:74 files removed 2024-04-16T07:06:57.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-04-16T07:06:58.831 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 133 kB/s | 77 kB 00:00 2024-04-16T07:06:58.838 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 152 kB/s | 77 kB 00:00 2024-04-16T07:06:59.490 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 19 kB/s | 11 kB 00:00 2024-04-16T07:06:59.491 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 19 kB/s | 11 kB 00:00 2024-04-16T07:06:59.948 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 4.4 kB/s | 1.8 kB 00:00 2024-04-16T07:07:00.108 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 3.3 kB/s | 1.8 kB 00:00 2024-04-16T07:07:00.870 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-04-16T07:07:01.245 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 6.3 MB/s | 6.7 MB 00:01 2024-04-16T07:07:02.894 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-04-16T07:07:02.971 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-04-16T07:07:04.630 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-04-16T07:07:05.212 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-16T07:07:07.940 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-16T07:07:08.540 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-04-16T07:07:13.507 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-04-16T07:07:15.658 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-04-16T07:07:17.823 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T07:07:21.917 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 671 kB/s | 6.2 MB 00:09 2024-04-16T07:07:23.725 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-16T07:07:23.759 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-16T07:07:25.278 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-16T07:07:26.767 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout:Install 22 Packages 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 58 M 2024-04-16T07:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:07:27.802 INFO:teuthology.orchestra.run.smithi104.stdout:(1/24): ceph-base-17.2.7-847.g45317f95.el9.x86_ 7.9 MB/s | 5.0 MB 00:00 2024-04-16T07:07:27.843 INFO:teuthology.orchestra.run.smithi104.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 621 kB/s | 25 kB 00:00 2024-04-16T07:07:27.918 INFO:teuthology.orchestra.run.smithi104.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 8.4 MB/s | 639 kB 00:00 2024-04-16T07:07:27.977 INFO:teuthology.orchestra.run.smithi104.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 7.8 MB/s | 457 kB 00:00 2024-04-16T07:07:28.294 INFO:teuthology.orchestra.run.smithi104.stdout:(5/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 16 MB/s | 4.9 MB 00:00 2024-04-16T07:07:28.344 INFO:teuthology.orchestra.run.smithi104.stdout:(6/24): python3-ceph-argparse-17.2.7-847.g45317 923 kB/s | 45 kB 00:00 2024-04-16T07:07:28.411 INFO:teuthology.orchestra.run.smithi104.stdout:(7/24): python3-ceph-common-17.2.7-847.g45317f9 1.7 MB/s | 117 kB 00:00 2024-04-16T07:07:28.486 INFO:teuthology.orchestra.run.smithi104.stdout:(8/24): python3-cephfs-17.2.7-847.g45317f95.el9 2.1 MB/s | 156 kB 00:00 2024-04-16T07:07:28.578 INFO:teuthology.orchestra.run.smithi104.stdout:(9/24): python3-rados-17.2.7-847.g45317f95.el9. 3.5 MB/s | 320 kB 00:00 2024-04-16T07:07:28.662 INFO:teuthology.orchestra.run.smithi104.stdout:(10/24): python3-rbd-17.2.7-847.g45317f95.el9.x 3.5 MB/s | 293 kB 00:00 2024-04-16T07:07:28.737 INFO:teuthology.orchestra.run.smithi104.stdout:(11/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.3 MB/s | 100 kB 00:00 2024-04-16T07:07:28.904 INFO:teuthology.orchestra.run.smithi104.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 960 kB/s | 159 kB 00:00 2024-04-16T07:07:28.946 INFO:teuthology.orchestra.run.smithi104.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-04-16T07:07:29.021 INFO:teuthology.orchestra.run.smithi104.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 661 kB 00:00 2024-04-16T07:07:29.063 INFO:teuthology.orchestra.run.smithi104.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-04-16T07:07:29.097 INFO:teuthology.orchestra.run.smithi104.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T07:07:29.656 INFO:teuthology.orchestra.run.smithi104.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 554 kB/s | 309 kB 00:00 2024-04-16T07:07:29.922 INFO:teuthology.orchestra.run.smithi104.stdout:(18/24): ceph-common-17.2.7-847.g45317f95.el9.x 8.2 MB/s | 22 MB 00:02 2024-04-16T07:07:29.947 INFO:teuthology.orchestra.run.smithi104.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 167 kB/s | 49 kB 00:00 2024-04-16T07:07:30.156 INFO:teuthology.orchestra.run.smithi104.stdout:(20/24): ceph-radosgw-17.2.7-847.g45317f95.el9. 4.7 MB/s | 14 MB 00:02 2024-04-16T07:07:30.348 INFO:teuthology.orchestra.run.smithi104.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-04-16T07:07:30.486 INFO:teuthology.orchestra.run.smithi104.stdout:(22/24): librados2-17.2.7-847.g45317f95.el9.x86 9.4 MB/s | 3.1 MB 00:00 2024-04-16T07:07:30.515 INFO:teuthology.orchestra.run.smithi104.stdout:(23/24): libunwind-1.6.2-1.el9.x86_64.rpm 114 kB/s | 67 kB 00:00 2024-04-16T07:07:30.724 INFO:teuthology.orchestra.run.smithi104.stdout:(24/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 7.8 MB/s | 2.9 MB 00:00 2024-04-16T07:07:30.730 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:07:30.731 INFO:teuthology.orchestra.run.smithi104.stdout:Total 15 MB/s | 58 MB 00:03 2024-04-16T07:07:30.732 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:07:30.805 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:07:30.805 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:07:31.342 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:07:31.342 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-16T07:07:32.701 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:07:33.102 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T07:07:33.253 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:07:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:07:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T07:07:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:07:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:07:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T07:07:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T07:07:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T07:07:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T07:07:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T07:07:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T07:07:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T07:07:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T07:07:34.604 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T07:07:34.614 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T07:07:34.615 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout:Install 22 Packages 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-04-16T07:07:34.616 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:07:34.617 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 58 M 2024-04-16T07:07:34.617 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:07:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T07:07:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:07:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:07:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T07:07:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T07:07:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T07:07:35.345 INFO:teuthology.orchestra.run.smithi026.stdout:(1/24): ceph-base-17.2.7-847.g45317f95.el9.x86_ 8.0 MB/s | 5.0 MB 00:00 2024-04-16T07:07:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T07:07:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T07:07:35.395 INFO:teuthology.orchestra.run.smithi026.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 514 kB/s | 25 kB 00:00 2024-04-16T07:07:35.479 INFO:teuthology.orchestra.run.smithi026.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 7.6 MB/s | 639 kB 00:00 2024-04-16T07:07:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T07:07:35.546 INFO:teuthology.orchestra.run.smithi026.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 6.8 MB/s | 457 kB 00:00 2024-04-16T07:07:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T07:07:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T07:07:35.888 INFO:teuthology.orchestra.run.smithi026.stdout:(5/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 14 MB/s | 4.9 MB 00:00 2024-04-16T07:07:35.938 INFO:teuthology.orchestra.run.smithi026.stdout:(6/24): python3-ceph-argparse-17.2.7-847.g45317 921 kB/s | 45 kB 00:00 2024-04-16T07:07:36.338 INFO:teuthology.orchestra.run.smithi026.stdout:(7/24): python3-ceph-common-17.2.7-847.g45317f9 295 kB/s | 117 kB 00:00 2024-04-16T07:07:36.405 INFO:teuthology.orchestra.run.smithi026.stdout:(8/24): python3-cephfs-17.2.7-847.g45317f95.el9 2.3 MB/s | 156 kB 00:00 2024-04-16T07:07:36.480 INFO:teuthology.orchestra.run.smithi026.stdout:(9/24): python3-rados-17.2.7-847.g45317f95.el9. 4.2 MB/s | 320 kB 00:00 2024-04-16T07:07:36.555 INFO:teuthology.orchestra.run.smithi026.stdout:(10/24): python3-rbd-17.2.7-847.g45317f95.el9.x 3.9 MB/s | 293 kB 00:00 2024-04-16T07:07:36.622 INFO:teuthology.orchestra.run.smithi026.stdout:(11/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.5 MB/s | 100 kB 00:00 2024-04-16T07:07:36.789 INFO:teuthology.orchestra.run.smithi026.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 962 kB/s | 159 kB 00:00 2024-04-16T07:07:36.831 INFO:teuthology.orchestra.run.smithi026.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-04-16T07:07:36.898 INFO:teuthology.orchestra.run.smithi026.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-04-16T07:07:36.940 INFO:teuthology.orchestra.run.smithi026.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-04-16T07:07:36.974 INFO:teuthology.orchestra.run.smithi026.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T07:07:37.283 INFO:teuthology.orchestra.run.smithi026.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-16T07:07:37.349 INFO:teuthology.orchestra.run.smithi026.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 739 kB/s | 49 kB 00:00 2024-04-16T07:07:37.399 INFO:teuthology.orchestra.run.smithi026.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-16T07:07:37.700 INFO:teuthology.orchestra.run.smithi026.stdout:(20/24): ceph-common-17.2.7-847.g45317f95.el9.x 7.5 MB/s | 22 MB 00:02 2024-04-16T07:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-04-16T07:07:38.001 INFO:teuthology.orchestra.run.smithi026.stdout:(22/24): ceph-radosgw-17.2.7-847.g45317f95.el9. 4.3 MB/s | 14 MB 00:03 2024-04-16T07:07:38.143 INFO:teuthology.orchestra.run.smithi026.stdout:(23/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 8.4 MB/s | 2.9 MB 00:00 2024-04-16T07:07:38.217 INFO:teuthology.orchestra.run.smithi026.stdout:(24/24): librados2-17.2.7-847.g45317f95.el9.x86 6.1 MB/s | 3.1 MB 00:00 2024-04-16T07:07:38.227 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:07:38.227 INFO:teuthology.orchestra.run.smithi026.stdout:Total 16 MB/s | 58 MB 00:03 2024-04-16T07:07:38.228 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:07:38.306 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:07:38.307 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:07:38.429 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:38.843 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:07:38.843 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:07:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:39.147 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T07:07:39.330 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T07:07:39.330 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:07:39.331 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:07:39.331 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:39.372 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:07:40.221 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:07:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T07:07:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:07:40.811 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:07:40.909 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T07:07:40.938 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:07:41.069 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:07:41.129 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T07:07:41.161 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T07:07:41.715 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T07:07:41.741 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T07:07:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T07:07:41.834 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T07:07:41.879 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T07:07:41.932 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T07:07:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T07:07:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:07:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:07:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T07:07:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T07:07:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T07:07:42.717 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T07:07:42.747 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T07:07:42.870 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T07:07:42.970 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T07:07:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T07:07:45.845 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:45.879 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:46.526 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T07:07:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T07:07:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T07:07:46.726 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:07:46.727 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:07:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:07:46.767 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:07:50.035 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-04-16T07:07:50.036 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:51.435 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T07:07:51.572 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T07:07:51.573 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:07:51.573 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:07:51.573 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:07:51.573 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:07:51.573 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:51.577 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T07:07:51.706 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T07:07:51.706 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T07:07:53.318 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T07:07:53.319 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T07:07:54.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:07:54.032 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:54.033 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:07:54.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-04-16T07:07:55.346 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:32 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:07:55.508 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:07:55.509 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:07:55.510 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:07:55.511 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-04-16T07:07:55.511 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:07:55.511 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 49 M 2024-04-16T07:07:55.511 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 193 M 2024-04-16T07:07:55.511 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:07:56.082 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 575 kB/s | 217 kB 00:00 2024-04-16T07:07:56.215 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 300 kB 00:00 2024-04-16T07:07:56.307 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 686 kB/s | 63 kB 00:00 2024-04-16T07:07:56.332 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 259 kB/s | 163 kB 00:00 2024-04-16T07:07:56.391 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-04-16T07:07:58.165 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:07:59.357 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 13 MB/s | 48 MB 00:03 2024-04-16T07:07:59.363 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:07:59.364 INFO:teuthology.orchestra.run.smithi104.stdout:Total 13 MB/s | 49 MB 00:03 2024-04-16T07:07:59.405 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:07:59.458 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:07:59.459 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:07:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T07:07:59.760 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T07:07:59.761 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:07:59.761 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:07:59.761 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:07:59.761 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:07:59.761 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:07:59.765 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T07:07:59.917 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:07:59.918 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:07:59.936 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T07:07:59.936 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:08:00.613 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:00.717 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T07:08:00.763 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T07:08:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T07:08:00.876 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T07:08:00.902 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T07:08:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T07:08:01.705 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T07:08:02.588 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T07:08:02.588 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:02.588 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-04-16T07:08:02.588 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:08:02.589 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:02.590 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:02.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-04-16T07:08:03.891 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:32 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:04.066 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:04.067 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:04.067 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-04-16T07:08:04.068 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:04.069 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 49 M 2024-04-16T07:08:04.069 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 193 M 2024-04-16T07:08:04.069 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:04.660 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 832 kB/s | 217 kB 00:00 2024-04-16T07:08:04.718 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 300 kB 00:00 2024-04-16T07:08:04.760 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T07:08:04.910 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 318 kB/s | 163 kB 00:00 2024-04-16T07:08:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 683 kB/s | 187 kB 00:00 2024-04-16T07:08:05.695 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T07:08:05.784 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T07:08:06.886 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 19 MB/s | 48 MB 00:02 2024-04-16T07:08:06.893 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:06.894 INFO:teuthology.orchestra.run.smithi026.stdout:Total 17 MB/s | 49 MB 00:02 2024-04-16T07:08:06.937 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:06.996 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:06.996 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:07.462 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:07.462 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:08.193 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T07:08:08.194 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T07:08:08.194 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T07:08:08.194 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T07:08:08.194 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T07:08:08.194 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T07:08:08.247 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T07:08:08.294 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T07:08:08.349 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T07:08:08.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T07:08:08.447 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:08:08.561 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:08:08.562 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:08.562 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:08.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-04-16T07:08:09.773 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:46 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:09.929 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:09.932 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-04-16T07:08:09.932 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:09.932 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-04-16T07:08:09.932 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:09.932 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:08:09.933 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:08:09.934 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:08:09.935 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout:Install 33 Packages 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:09.936 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 27 M 2024-04-16T07:08:09.937 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 96 M 2024-04-16T07:08:09.937 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:10.702 INFO:teuthology.orchestra.run.smithi104.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 25 kB/s | 6.5 kB 00:00 2024-04-16T07:08:10.961 INFO:teuthology.orchestra.run.smithi104.stdout:(2/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 4.0 MB/s | 2.1 MB 00:00 2024-04-16T07:08:11.027 INFO:teuthology.orchestra.run.smithi104.stdout:(3/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 2.4 MB/s | 1.4 MB 00:00 2024-04-16T07:08:11.191 INFO:teuthology.orchestra.run.smithi104.stdout:(4/33): ceph-mgr-modules-core-17.2.7-847.g45317 1.4 MB/s | 240 kB 00:00 2024-04-16T07:08:11.345 INFO:teuthology.orchestra.run.smithi104.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 950 kB/s | 144 kB 00:00 2024-04-16T07:08:11.378 INFO:teuthology.orchestra.run.smithi104.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 969 kB/s | 32 kB 00:00 2024-04-16T07:08:11.412 INFO:teuthology.orchestra.run.smithi104.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T07:08:11.445 INFO:teuthology.orchestra.run.smithi104.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 39 kB 00:00 2024-04-16T07:08:11.496 INFO:teuthology.orchestra.run.smithi104.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-04-16T07:08:11.563 INFO:teuthology.orchestra.run.smithi104.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-16T07:08:11.604 INFO:teuthology.orchestra.run.smithi104.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 103 kB 00:00 2024-04-16T07:08:11.646 INFO:teuthology.orchestra.run.smithi104.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.0 MB/s | 124 kB 00:00 2024-04-16T07:08:11.688 INFO:teuthology.orchestra.run.smithi104.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.5 MB/s | 187 kB 00:00 2024-04-16T07:08:11.780 INFO:teuthology.orchestra.run.smithi104.stdout:(14/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_ 3.6 MB/s | 3.9 MB 00:01 2024-04-16T07:08:11.989 INFO:teuthology.orchestra.run.smithi104.stdout:(15/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 15 MB/s | 15 MB 00:01 2024-04-16T07:08:12.164 INFO:teuthology.orchestra.run.smithi104.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 263 kB/s | 125 kB 00:00 2024-04-16T07:08:12.256 INFO:teuthology.orchestra.run.smithi104.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 91 kB/s | 43 kB 00:00 2024-04-16T07:08:12.340 INFO:teuthology.orchestra.run.smithi104.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-04-16T07:08:12.365 INFO:teuthology.orchestra.run.smithi104.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 100 kB/s | 11 kB 00:00 2024-04-16T07:08:12.390 INFO:teuthology.orchestra.run.smithi104.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 516 kB/s | 172 kB 00:00 2024-04-16T07:08:12.424 INFO:teuthology.orchestra.run.smithi104.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 310 kB/s | 18 kB 00:00 2024-04-16T07:08:12.449 INFO:teuthology.orchestra.run.smithi104.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 282 kB/s | 23 kB 00:00 2024-04-16T07:08:12.466 INFO:teuthology.orchestra.run.smithi104.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-04-16T07:08:12.491 INFO:teuthology.orchestra.run.smithi104.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 299 kB/s | 20 kB 00:00 2024-04-16T07:08:12.525 INFO:teuthology.orchestra.run.smithi104.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 809 kB/s | 46 kB 00:00 2024-04-16T07:08:12.550 INFO:teuthology.orchestra.run.smithi104.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 958 kB/s | 79 kB 00:00 2024-04-16T07:08:12.592 INFO:teuthology.orchestra.run.smithi104.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 250 kB/s | 16 kB 00:00 2024-04-16T07:08:12.625 INFO:teuthology.orchestra.run.smithi104.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-16T07:08:12.651 INFO:teuthology.orchestra.run.smithi104.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 913 kB/s | 90 kB 00:00 2024-04-16T07:08:12.667 INFO:teuthology.orchestra.run.smithi104.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 483 kB/s | 36 kB 00:00 2024-04-16T07:08:12.718 INFO:teuthology.orchestra.run.smithi104.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-04-16T07:08:12.743 INFO:teuthology.orchestra.run.smithi104.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 274 kB/s | 20 kB 00:00 2024-04-16T07:08:12.777 INFO:teuthology.orchestra.run.smithi104.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-04-16T07:08:12.787 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:12.788 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.5 MB/s | 27 MB 00:02 2024-04-16T07:08:12.829 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:12.913 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:12.914 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:13.344 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:13.345 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T07:08:13.692 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T07:08:14.701 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:14.764 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T07:08:14.799 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T07:08:14.844 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T07:08:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T07:08:15.055 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T07:08:15.128 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T07:08:15.221 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T07:08:15.289 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T07:08:15.332 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T07:08:15.398 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T07:08:15.475 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T07:08:15.732 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T07:08:15.794 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T07:08:15.839 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T07:08:15.887 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T07:08:15.915 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T07:08:15.923 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T07:08:16.028 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T07:08:16.073 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T07:08:16.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T07:08:16.157 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T07:08:16.212 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T07:08:16.253 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:16.280 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T07:08:16.439 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T07:08:16.544 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T07:08:16.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-04-16T07:08:16.728 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T07:08:16.900 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T07:08:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T07:08:17.156 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T07:08:17.245 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T07:08:17.245 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:17.245 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:08:17.245 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:08:17.246 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:08:17.246 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:17.508 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:46 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:17.674 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T07:08:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:08:17.674 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T07:08:17.674 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:08:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:08:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout:Install 33 Packages 2024-04-16T07:08:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:17.678 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 27 M 2024-04-16T07:08:17.678 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 96 M 2024-04-16T07:08:17.678 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:18.172 INFO:teuthology.orchestra.run.smithi026.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 26 kB/s | 6.5 kB 00:00 2024-04-16T07:08:18.547 INFO:teuthology.orchestra.run.smithi026.stdout:(2/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_6 10 MB/s | 3.9 MB 00:00 2024-04-16T07:08:18.699 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T07:08:18.705 INFO:teuthology.orchestra.run.smithi026.stdout:(3/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 1.8 MB/s | 1.4 MB 00:00 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:08:18.779 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:18.781 INFO:teuthology.orchestra.run.smithi026.stdout:(4/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 2.4 MB/s | 2.1 MB 00:00 2024-04-16T07:08:18.823 INFO:teuthology.orchestra.run.smithi026.stdout:(5/33): ceph-mgr-modules-core-17.2.7-847.g45317 2.0 MB/s | 240 kB 00:00 2024-04-16T07:08:18.998 INFO:teuthology.orchestra.run.smithi026.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 298 kB/s | 32 kB 00:00 2024-04-16T07:08:19.023 INFO:teuthology.orchestra.run.smithi026.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 598 kB/s | 144 kB 00:00 2024-04-16T07:08:19.049 INFO:teuthology.orchestra.run.smithi026.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 958 kB/s | 47 kB 00:00 2024-04-16T07:08:19.074 INFO:teuthology.orchestra.run.smithi026.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 778 kB/s | 39 kB 00:00 2024-04-16T07:08:19.110 INFO:teuthology.orchestra.run.smithi026.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.9 MB/s | 241 kB 00:00 2024-04-16T07:08:19.183 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T07:08:19.219 INFO:teuthology.orchestra.run.smithi026.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 8.1 MB/s | 1.1 MB 00:00 2024-04-16T07:08:19.244 INFO:teuthology.orchestra.run.smithi026.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 795 kB/s | 103 kB 00:00 2024-04-16T07:08:19.254 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T07:08:19.254 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:19.254 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:08:19.254 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:08:19.255 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:08:19.255 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:19.269 INFO:teuthology.orchestra.run.smithi026.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.5 MB/s | 124 kB 00:00 2024-04-16T07:08:19.324 INFO:teuthology.orchestra.run.smithi026.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 187 kB 00:00 2024-04-16T07:08:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T07:08:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T07:08:19.535 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:19.535 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:08:19.535 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:08:19.535 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:08:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:19.609 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T07:08:19.620 INFO:teuthology.orchestra.run.smithi026.stdout:(15/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 14 MB/s | 15 MB 00:01 2024-04-16T07:08:19.645 INFO:teuthology.orchestra.run.smithi026.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 136 kB/s | 43 kB 00:00 2024-04-16T07:08:19.738 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T07:08:19.738 INFO:teuthology.orchestra.run.smithi104.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T07:08:19.738 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:19.745 INFO:teuthology.orchestra.run.smithi026.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-16T07:08:19.779 INFO:teuthology.orchestra.run.smithi026.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-04-16T07:08:19.804 INFO:teuthology.orchestra.run.smithi026.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 193 kB/s | 11 kB 00:00 2024-04-16T07:08:19.829 INFO:teuthology.orchestra.run.smithi026.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 360 kB/s | 18 kB 00:00 2024-04-16T07:08:19.855 INFO:teuthology.orchestra.run.smithi026.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 470 kB/s | 23 kB 00:00 2024-04-16T07:08:19.880 INFO:teuthology.orchestra.run.smithi026.stdout:(22/33): python3-requests-2.25.1-8.el9.noarch.r 205 kB/s | 125 kB 00:00 2024-04-16T07:08:19.897 INFO:teuthology.orchestra.run.smithi026.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-04-16T07:08:19.913 INFO:teuthology.orchestra.run.smithi026.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 342 kB/s | 20 kB 00:00 2024-04-16T07:08:19.939 INFO:teuthology.orchestra.run.smithi026.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-04-16T07:08:19.964 INFO:teuthology.orchestra.run.smithi026.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-04-16T07:08:19.998 INFO:teuthology.orchestra.run.smithi026.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-16T07:08:20.015 INFO:teuthology.orchestra.run.smithi026.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 352 kB/s | 16 kB 00:00 2024-04-16T07:08:20.040 INFO:teuthology.orchestra.run.smithi026.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-16T07:08:20.065 INFO:teuthology.orchestra.run.smithi026.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 550 kB/s | 36 kB 00:00 2024-04-16T07:08:20.174 INFO:teuthology.orchestra.run.smithi026.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-16T07:08:20.199 INFO:teuthology.orchestra.run.smithi026.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 152 kB/s | 20 kB 00:00 2024-04-16T07:08:20.249 INFO:teuthology.orchestra.run.smithi026.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2024-04-16T07:08:20.261 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:20.261 INFO:teuthology.orchestra.run.smithi026.stdout:Total 10 MB/s | 27 MB 00:02 2024-04-16T07:08:20.305 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:20.397 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:20.398 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:20.816 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:20.816 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T07:08:21.358 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T07:08:21.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T07:08:21.360 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T07:08:22.168 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:22.244 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T07:08:22.279 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T07:08:22.321 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T07:08:22.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:08:22.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:08:22.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:22.394 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T07:08:22.580 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T07:08:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T07:08:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T07:08:22.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-04-16T07:08:22.797 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T07:08:22.839 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T07:08:22.906 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T07:08:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T07:08:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T07:08:23.302 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T07:08:23.354 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T07:08:23.399 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T07:08:23.447 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T07:08:23.561 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T07:08:23.594 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:23.605 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T07:08:23.638 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T07:08:23.690 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T07:08:23.696 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:08:23.749 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:23.750 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:08:23.750 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:23.754 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T07:08:23.793 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T07:08:23.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-04-16T07:08:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T07:08:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T07:08:24.096 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T07:08:24.282 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T07:08:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T07:08:24.627 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:01 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:24.784 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 82 k 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 372 k 2024-04-16T07:08:24.785 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:24.813 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T07:08:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T07:08:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T07:08:24.930 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:24.930 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:08:24.931 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:08:24.931 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:08:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:25.279 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 166 kB/s | 82 kB 00:00 2024-04-16T07:08:25.280 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:25.281 INFO:teuthology.orchestra.run.smithi104.stdout:Total 165 kB/s | 82 kB 00:00 2024-04-16T07:08:25.282 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:25.299 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:25.299 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:25.306 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:26.091 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:26.208 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:08:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T07:08:26.991 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:26.991 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:26.992 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:26.992 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:26.992 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:26.992 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T07:08:27.032 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:27.032 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:08:27.032 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:08:27.032 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:08:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:27.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:08:27.258 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:08:27.340 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T07:08:27.556 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T07:08:27.556 INFO:teuthology.orchestra.run.smithi026.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T07:08:27.557 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:28.106 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:05 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:28.263 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:08:28.264 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:28.265 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 137 k 2024-04-16T07:08:28.265 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 379 k 2024-04-16T07:08:28.265 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:28.642 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 365 kB/s | 137 kB 00:00 2024-04-16T07:08:28.643 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:28.644 INFO:teuthology.orchestra.run.smithi104.stdout:Total 362 kB/s | 137 kB 00:00 2024-04-16T07:08:28.645 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:28.766 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:28.767 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:28.964 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:29.026 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:29.100 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:29.100 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:29.100 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:08:29.100 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T07:08:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T07:08:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T07:08:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T07:08:30.361 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:30.361 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:30.362 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:30.362 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.362 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:30.362 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T07:08:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:30.382 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:08:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:08:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:30.385 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:30.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-04-16T07:08:30.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-04-16T07:08:31.504 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:31.606 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:08:31.647 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:31.659 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:31.660 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:08:31.661 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:31.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:08:31.756 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:08:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:31.814 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:08:31.814 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:31.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-04-16T07:08:32.580 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:32.737 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:08:32.738 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout:Install 7 Packages 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:32.739 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.7 M 2024-04-16T07:08:32.740 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 80 M 2024-04-16T07:08:32.740 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:32.763 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:01 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:32.925 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:32.926 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:32.926 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:08:32.926 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:32.926 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 82 k 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 372 k 2024-04-16T07:08:32.927 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:33.265 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 243 kB/s | 82 kB 00:00 2024-04-16T07:08:33.266 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:33.267 INFO:teuthology.orchestra.run.smithi026.stdout:Total 241 kB/s | 82 kB 00:00 2024-04-16T07:08:33.268 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:33.276 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:33.276 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:33.278 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:33.279 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:33.386 INFO:teuthology.orchestra.run.smithi104.stdout:(1/7): ceph-prometheus-alerts-17.2.7-847.g45317 52 kB/s | 15 kB 00:00 2024-04-16T07:08:33.400 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:33.428 INFO:teuthology.orchestra.run.smithi104.stdout:(2/7): ceph-grafana-dashboards-17.2.7-847.g4531 75 kB/s | 24 kB 00:00 2024-04-16T07:08:33.645 INFO:teuthology.orchestra.run.smithi104.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 35 kB/s | 9.0 kB 00:00 2024-04-16T07:08:33.762 INFO:teuthology.orchestra.run.smithi104.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 265 kB/s | 31 kB 00:00 2024-04-16T07:08:33.787 INFO:teuthology.orchestra.run.smithi104.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 114 kB/s | 41 kB 00:00 2024-04-16T07:08:33.945 INFO:teuthology.orchestra.run.smithi104.stdout:(6/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 4.0 MB/s | 3.4 MB 00:00 2024-04-16T07:08:33.979 INFO:teuthology.orchestra.run.smithi104.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 873 kB/s | 188 kB 00:00 2024-04-16T07:08:33.987 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:33.988 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.9 MB/s | 3.7 MB 00:01 2024-04-16T07:08:33.989 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:34.015 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:34.015 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:34.059 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:34.145 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:34.146 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:34.190 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T07:08:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T07:08:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T07:08:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T07:08:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:35.113 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:35.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:08:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T07:08:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T07:08:36.252 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:05 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:36.416 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:36.417 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 137 k 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 379 k 2024-04-16T07:08:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:36.804 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 356 kB/s | 137 kB 00:00 2024-04-16T07:08:36.806 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:36.807 INFO:teuthology.orchestra.run.smithi026.stdout:Total 353 kB/s | 137 kB 00:00 2024-04-16T07:08:36.807 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:36.834 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:36.834 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:36.917 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:36.917 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:37.112 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:37.228 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T07:08:37.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T07:08:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:37.263 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:08:37.264 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:08:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:37.591 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T07:08:37.592 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T07:08:37.592 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:08:37.592 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:08:37.592 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:37.592 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:37.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:08:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T07:08:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:38.554 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:08:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:38.555 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:38.709 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:15 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:38.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-04-16T07:08:38.871 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:08:38.872 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-04-16T07:08:38.873 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:38.874 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 37 M 2024-04-16T07:08:38.874 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 209 M 2024-04-16T07:08:38.874 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:39.587 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 121 kB/s | 32 kB 00:00 2024-04-16T07:08:39.662 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 234 kB/s | 17 kB 00:00 2024-04-16T07:08:39.729 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 519 kB/s | 34 kB 00:00 2024-04-16T07:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:39.869 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:08:39.925 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:39.926 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:08:39.926 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:39.963 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-04-16T07:08:40.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:08:40.088 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-84 9.7 MB/s | 7.4 MB 00:00 2024-04-16T07:08:40.230 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.5 MB/s | 4.7 MB 00:00 2024-04-16T07:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 694 kB/s | 217 kB 00:00 2024-04-16T07:08:40.539 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.4 MB/s | 4.8 MB 00:00 2024-04-16T07:08:40.893 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:41.062 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:08:41.062 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout:Install 7 Packages 2024-04-16T07:08:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:41.064 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.7 M 2024-04-16T07:08:41.064 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 80 M 2024-04-16T07:08:41.064 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:41.272 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-16T07:08:41.306 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 260 kB/s | 195 kB 00:00 2024-04-16T07:08:41.381 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 837 kB/s | 803 kB 00:00 2024-04-16T07:08:41.432 INFO:teuthology.orchestra.run.smithi026.stdout:(1/7): ceph-prometheus-alerts-17.2.7-847.g45317 58 kB/s | 15 kB 00:00 2024-04-16T07:08:41.566 INFO:teuthology.orchestra.run.smithi026.stdout:(2/7): ceph-grafana-dashboards-17.2.7-847.g4531 63 kB/s | 24 kB 00:00 2024-04-16T07:08:41.675 INFO:teuthology.orchestra.run.smithi026.stdout:(3/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 6.8 MB/s | 3.4 MB 00:00 2024-04-16T07:08:41.681 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 602 kB/s | 245 kB 00:00 2024-04-16T07:08:41.687 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:41.687 INFO:teuthology.orchestra.run.smithi104.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-16T07:08:41.700 INFO:teuthology.orchestra.run.smithi026.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 34 kB/s | 9.0 kB 00:00 2024-04-16T07:08:41.805 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:41.858 INFO:teuthology.orchestra.run.smithi026.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 168 kB/s | 31 kB 00:00 2024-04-16T07:08:41.870 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:41.870 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:41.917 INFO:teuthology.orchestra.run.smithi026.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 117 kB/s | 41 kB 00:00 2024-04-16T07:08:42.200 INFO:teuthology.orchestra.run.smithi026.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 377 kB/s | 188 kB 00:00 2024-04-16T07:08:42.205 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:42.206 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.2 MB/s | 3.7 MB 00:01 2024-04-16T07:08:42.207 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:42.235 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:42.235 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:42.382 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:42.383 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:42.389 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:42.389 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:42.800 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:42.877 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T07:08:42.925 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T07:08:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T07:08:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T07:08:43.117 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T07:08:43.219 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:43.319 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:08:43.354 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:08:43.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:08:43.541 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:08:43.806 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:08:43.838 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:08:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:08:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T07:08:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T07:08:44.768 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:08:44.905 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T07:08:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:08:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:08:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:45.971 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:46.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:08:46.412 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:08:47.121 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:16 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:08:47.289 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:08:47.290 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-04-16T07:08:47.290 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:08:47.290 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:08:47.291 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 37 M 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-04-16T07:08:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:08:47.550 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:08:47.599 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T07:08:48.151 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 138 kB/s | 32 kB 00:00 2024-04-16T07:08:48.193 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 463 kB/s | 17 kB 00:00 2024-04-16T07:08:48.293 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.0 MB/s | 3.0 MB 00:00 2024-04-16T07:08:48.318 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 317 kB/s | 34 kB 00:00 2024-04-16T07:08:48.569 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-04-16T07:08:48.669 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-04-16T07:08:48.727 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2024-04-16T07:08:49.220 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-16T07:08:49.269 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-16T07:08:49.420 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 981 kB/s | 195 kB 00:00 2024-04-16T07:08:49.854 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 421 kB/s | 245 kB 00:00 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:08:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:08:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:08:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:08:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:08:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:08:50.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:08:50.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:50.385 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:50.385 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:50.386 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:50.546 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 2.8 MB/s | 7.4 MB 00:02 2024-04-16T07:08:50.551 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:50.551 INFO:teuthology.orchestra.run.smithi026.stdout:Total 11 MB/s | 37 MB 00:03 2024-04-16T07:08:50.671 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:08:50.736 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:08:50.736 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:08:50.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:08:51.269 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:08:51.269 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:08:51.657 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:51.817 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:08:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-04-16T07:08:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:51.821 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.0 M 2024-04-16T07:08:51.821 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-04-16T07:08:51.821 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:52.077 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 157 kB/s | 24 kB 00:00 2024-04-16T07:08:52.102 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 100 kB/s | 18 kB 00:00 2024-04-16T07:08:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:08:52.152 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T07:08:52.178 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T07:08:52.209 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:08:52.211 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 211 kB 00:00 2024-04-16T07:08:52.236 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 762 kB/s | 43 kB 00:00 2024-04-16T07:08:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:08:52.262 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 139 kB/s | 47 kB 00:00 2024-04-16T07:08:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:08:52.438 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:08:52.587 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-04-16T07:08:52.612 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 81 kB/s | 32 kB 00:00 2024-04-16T07:08:52.713 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:08:52.745 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:08:52.746 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 446 kB/s | 59 kB 00:00 2024-04-16T07:08:52.788 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 386 kB/s | 203 kB 00:00 2024-04-16T07:08:52.855 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:08:52.880 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 674 kB/s | 90 kB 00:00 2024-04-16T07:08:53.005 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.5 MB/s | 1.0 MB 00:00 2024-04-16T07:08:53.015 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:53.016 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-04-16T07:08:53.017 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:53.057 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:53.057 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:53.213 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:53.213 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:08:53.744 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:08:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:08:53.889 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:08:54.117 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:08:54.182 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:08:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:08:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:08:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:08:54.474 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:08:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:08:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:08:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:08:55.732 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:08:55.782 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:08:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:08:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:08:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:08:56.771 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:08:56.772 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:08:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:08:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:08:57.224 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:08:57.225 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:08:57.225 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:08:57.225 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:08:57.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:57.225 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:08:57.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:08:58.403 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:35 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:08:58.566 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:08:58.567 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:08:58.568 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-04-16T07:08:58.569 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-04-16T07:08:58.569 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:08:58.569 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 6.8 M 2024-04-16T07:08:58.569 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-04-16T07:08:58.569 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:08:59.005 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 943 kB/s | 248 kB 00:00 2024-04-16T07:08:59.139 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-16T07:08:59.273 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 233 kB/s | 123 kB 00:00 2024-04-16T07:08:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T07:08:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T07:08:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:08:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:08:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:08:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:08:59.565 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-04-16T07:08:59.623 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 909 kB/s | 548 kB 00:00 2024-04-16T07:08:59.706 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 140 kB/s | 51 kB 00:00 2024-04-16T07:08:59.712 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:08:59.713 INFO:teuthology.orchestra.run.smithi104.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-04-16T07:08:59.748 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:08:59.768 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:08:59.768 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:08:59.802 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:08:59.802 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:59.802 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:08:59.803 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:08:59.902 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:08:59.903 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:00.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:09:00.286 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:00.527 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:09:00.598 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:09:01.132 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:01.299 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:09:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-04-16T07:09:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:01.303 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:09:01.303 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-04-16T07:09:01.303 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-04-16T07:09:01.303 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:01.434 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:09:01.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:09:01.536 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T07:09:01.602 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-04-16T07:09:01.627 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 124 kB/s | 24 kB 00:00 2024-04-16T07:09:01.678 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T07:09:01.703 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T07:09:01.736 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 211 kB 00:00 2024-04-16T07:09:01.762 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 143 kB/s | 47 kB 00:00 2024-04-16T07:09:01.787 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 528 kB/s | 43 kB 00:00 2024-04-16T07:09:02.054 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-04-16T07:09:02.079 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 95 kB/s | 32 kB 00:00 2024-04-16T07:09:02.298 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 399 kB/s | 203 kB 00:00 2024-04-16T07:09:02.330 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 237 kB/s | 59 kB 00:00 2024-04-16T07:09:02.413 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-04-16T07:09:02.438 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 646 kB/s | 90 kB 00:00 2024-04-16T07:09:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:02.451 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-04-16T07:09:02.452 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:02.495 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:02.495 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:09:02.517 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:09:02.662 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:02.662 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:02.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:02.883 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:03.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-04-16T07:09:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:09:03.616 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:09:03.690 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:09:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:09:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:09:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:09:04.052 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:41 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:04.058 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:09:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:09:04.217 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:09:04.218 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 891 k 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.6 M 2024-04-16T07:09:04.219 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:04.366 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 541 kB/s | 79 kB 00:00 2024-04-16T07:09:04.715 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 1.6 MB/s | 813 kB 00:00 2024-04-16T07:09:04.719 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:04.720 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.7 MB/s | 891 kB 00:00 2024-04-16T07:09:04.720 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:04.789 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:04.789 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:04.883 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:04.883 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:05.105 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:09:05.134 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:05.166 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:09:05.200 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:09:05.257 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:09:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T07:09:05.302 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:09:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T07:09:06.166 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:09:06.167 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:09:06.291 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T07:09:06.291 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T07:09:06.629 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:09:06.629 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:09:06.630 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:06.631 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:06.640 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:06.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:09:06.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-04-16T07:09:07.857 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:36 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:07.871 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:08.027 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:09:08.028 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.8 M 2024-04-16T07:09:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-04-16T07:09:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:08.033 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:08.033 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:08.033 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:09:08.034 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:08.035 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 126 k 2024-04-16T07:09:08.035 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 448 k 2024-04-16T07:09:08.035 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:08.312 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 458 kB/s | 126 kB 00:00 2024-04-16T07:09:08.313 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:08.314 INFO:teuthology.orchestra.run.smithi104.stdout:Total 451 kB/s | 126 kB 00:00 2024-04-16T07:09:08.315 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:08.333 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:08.334 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:08.446 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:08.446 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:08.644 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:08.762 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:08.931 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-16T07:09:08.964 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 301 kB/s | 123 kB 00:00 2024-04-16T07:09:09.048 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 503 kB/s | 248 kB 00:00 2024-04-16T07:09:09.115 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 386 kB/s | 58 kB 00:00 2024-04-16T07:09:09.223 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-04-16T07:09:09.240 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 267 kB/s | 51 kB 00:00 2024-04-16T07:09:09.243 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:09.244 INFO:teuthology.orchestra.run.smithi026.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-04-16T07:09:09.287 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:09.324 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:09.324 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:09.411 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:09.472 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:09.472 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:09.658 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:09.658 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:09.658 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:09.658 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:09.659 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:09.659 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:10.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-04-16T07:09:10.123 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:09:10.183 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:09:10.883 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:47 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:10.909 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:09:10.989 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:11.036 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:09:11.042 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:11.109 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:09:11.137 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T07:09:11.139 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-04-16T07:09:11.994 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:09:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:09:12.153 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:09:12.154 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:12.155 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 25 k 2024-04-16T07:09:12.155 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 110 k 2024-04-16T07:09:12.155 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:12.404 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:12.465 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 81 kB/s | 25 kB 00:00 2024-04-16T07:09:12.466 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:12.467 INFO:teuthology.orchestra.run.smithi104.stdout:Total 80 kB/s | 25 kB 00:00 2024-04-16T07:09:12.468 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:12.480 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:12.480 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:12.486 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:12.487 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:12.624 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:12.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-04-16T07:09:12.762 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:13.220 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:13.508 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:13.509 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:13.620 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:42 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:13.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-04-16T07:09:13.790 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:13.790 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T07:09:13.790 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:13.790 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-04-16T07:09:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:13.792 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 891 k 2024-04-16T07:09:13.792 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.6 M 2024-04-16T07:09:13.792 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:13.943 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 524 kB/s | 79 kB 00:00 2024-04-16T07:09:14.276 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 1.6 MB/s | 813 kB 00:00 2024-04-16T07:09:14.280 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:14.281 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.8 MB/s | 891 kB 00:00 2024-04-16T07:09:14.281 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:14.353 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:14.353 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:14.454 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:14.454 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:14.572 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:51 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:14.679 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:14.734 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:14.735 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:14.735 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:14.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-04-16T07:09:14.872 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:09:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T07:09:15.627 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:15.732 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:15.788 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:15.789 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:15.789 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:15.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-04-16T07:09:15.922 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T07:09:15.922 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:16.279 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:16.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-04-16T07:09:16.706 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:53 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:16.811 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:16.864 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:16.866 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:16.866 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:16.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-04-16T07:09:17.558 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:46 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:17.731 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:17.731 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T07:09:17.731 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:17.732 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 126 k 2024-04-16T07:09:17.733 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 448 k 2024-04-16T07:09:17.733 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:17.785 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:54 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:17.891 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:17.945 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:17.946 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:17.946 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:18.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-04-16T07:09:18.130 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 318 kB/s | 126 kB 00:00 2024-04-16T07:09:18.132 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:18.133 INFO:teuthology.orchestra.run.smithi026.stdout:Total 315 kB/s | 126 kB 00:00 2024-04-16T07:09:18.133 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:18.152 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:18.153 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:18.241 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:18.241 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:18.438 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:18.829 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:55 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:18.935 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:18.989 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:18.991 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:18.991 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:19.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-04-16T07:09:19.105 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:19.364 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:19.364 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:19.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-04-16T07:09:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:56 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:19.991 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:20.044 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:20.045 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-16T07:09:20.045 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:20.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-04-16T07:09:20.590 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:49 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:20.699 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:20.756 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:20.757 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:20.757 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:20.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-04-16T07:09:20.950 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:57 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:21.114 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:21.114 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:21.114 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:09:21.115 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:21.116 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 82 k 2024-04-16T07:09:21.116 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 222 k 2024-04-16T07:09:21.116 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:21.476 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 227 kB/s | 82 kB 00:00 2024-04-16T07:09:21.477 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:21.478 INFO:teuthology.orchestra.run.smithi104.stdout:Total 225 kB/s | 82 kB 00:00 2024-04-16T07:09:21.478 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:21.502 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:21.502 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:21.603 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:21.603 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:21.703 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:50 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:21.788 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:21.869 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:09:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:21.871 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 25 k 2024-04-16T07:09:21.871 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 110 k 2024-04-16T07:09:21.871 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:21.911 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:22.126 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 98 kB/s | 25 kB 00:00 2024-04-16T07:09:22.128 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:22.128 INFO:teuthology.orchestra.run.smithi026.stdout:Total 97 kB/s | 25 kB 00:00 2024-04-16T07:09:22.129 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:22.141 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:22.141 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:22.147 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:22.305 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:22.410 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:22.726 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:22.953 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:23.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-04-16T07:09:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:23.248 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:23.248 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:23.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-04-16T07:09:23.918 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:00 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:24.076 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T07:09:24.077 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.9 M 2024-04-16T07:09:24.078 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 12 M 2024-04-16T07:09:24.079 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:24.372 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:53 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:24.482 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:24.540 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:24.542 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:24.542 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:24.605 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 5.6 MB/s | 2.9 MB 00:00 2024-04-16T07:09:24.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-04-16T07:09:24.607 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:24.608 INFO:teuthology.orchestra.run.smithi104.stdout:Total 5.5 MB/s | 2.9 MB 00:00 2024-04-16T07:09:24.608 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:24.639 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:24.639 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:24.766 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:24.766 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:25.281 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:25.431 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:09:25.431 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:09:25.431 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:09:25.431 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:09:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:25.478 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:54 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:25.589 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:25.647 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:25.648 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:25.648 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:25.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-04-16T07:09:26.572 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:55 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:26.680 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:26.738 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:26.739 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:26.739 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:26.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-04-16T07:09:26.873 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:26.873 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:26.873 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:26.873 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:26.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:26.874 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:27.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-04-16T07:09:27.691 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:56 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:27.800 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:27.857 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:27.858 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:27.858 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:27.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-04-16T07:09:28.053 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:05 ago on Tue 16 Apr 2024 07:07:23 AM UTC. 2024-04-16T07:09:28.216 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-16T07:09:28.217 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-16T07:09:28.218 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-04-16T07:09:28.218 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:28.218 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 165 k 2024-04-16T07:09:28.218 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 477 k 2024-04-16T07:09:28.218 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-16T07:09:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 571 kB/s | 165 kB 00:00 2024-04-16T07:09:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:Total 564 kB/s | 165 kB 00:00 2024-04-16T07:09:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-16T07:09:28.536 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-16T07:09:28.536 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-16T07:09:28.636 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-16T07:09:28.637 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-16T07:09:28.774 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:57 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:28.849 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-16T07:09:28.883 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:28.941 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:28.943 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:28.943 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:28.975 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:29.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-04-16T07:09:29.523 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:29.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:29.851 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:29.852 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-16T07:09:29.852 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:29.852 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:09:29.852 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-16T07:09:29.952 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:58 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:30.061 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T07:09:30.117 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:30.119 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T07:09:30.119 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:30.181 DEBUG:teuthology.parallel:result is None 2024-04-16T07:09:30.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-04-16T07:09:31.071 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:00 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:31.241 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:09:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 82 k 2024-04-16T07:09:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 222 k 2024-04-16T07:09:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:31.470 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 361 kB/s | 82 kB 00:00 2024-04-16T07:09:31.471 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:31.472 INFO:teuthology.orchestra.run.smithi026.stdout:Total 356 kB/s | 82 kB 00:00 2024-04-16T07:09:31.473 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:31.497 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:31.497 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:31.601 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:31.601 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:31.861 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:32.395 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:32.757 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:33.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-04-16T07:09:33.936 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:02 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:34.102 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:34.103 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:34.104 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:09:34.104 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:34.104 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.9 M 2024-04-16T07:09:34.104 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 12 M 2024-04-16T07:09:34.104 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:34.891 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 3.7 MB/s | 2.9 MB 00:00 2024-04-16T07:09:34.903 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:34.903 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.7 MB/s | 2.9 MB 00:00 2024-04-16T07:09:34.904 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:34.927 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:34.927 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:35.058 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:35.058 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:35.594 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:35.650 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:35.740 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:09:35.740 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:09:35.740 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:09:35.740 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:09:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:37.266 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:37.267 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:37.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-04-16T07:09:38.508 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:07 ago on Tue 16 Apr 2024 07:07:31 AM UTC. 2024-04-16T07:09:38.679 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T07:09:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:38.681 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 165 k 2024-04-16T07:09:38.681 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 477 k 2024-04-16T07:09:38.681 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T07:09:38.927 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 672 kB/s | 165 kB 00:00 2024-04-16T07:09:38.929 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:09:38.929 INFO:teuthology.orchestra.run.smithi026.stdout:Total 663 kB/s | 165 kB 00:00 2024-04-16T07:09:38.930 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T07:09:38.957 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T07:09:38.958 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T07:09:39.059 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T07:09:39.060 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T07:09:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T07:09:39.414 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:39.975 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:40.330 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T07:09:40.331 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:40.331 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T07:09:40.331 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T07:09:40.331 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:09:40.331 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T07:09:40.618 DEBUG:teuthology.parallel:result is None 2024-04-16T07:09:40.618 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:09:40.619 INFO:teuthology.packaging:ref: None 2024-04-16T07:09:40.619 INFO:teuthology.packaging:tag: None 2024-04-16T07:09:40.619 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:09:40.619 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:09:40.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T07:09:40.777 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:09:40.849 INFO:teuthology.orchestra.run.smithi026.stdout:17.2.7-847.g45317f95.el9 2024-04-16T07:09:40.850 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T07:09:40.850 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T07:09:40.852 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:09:40.852 INFO:teuthology.packaging:ref: None 2024-04-16T07:09:40.852 INFO:teuthology.packaging:tag: None 2024-04-16T07:09:40.852 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T07:09:40.852 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:09:40.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T07:09:41.079 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:09:41.148 INFO:teuthology.orchestra.run.smithi104.stdout:17.2.7-847.g45317f95.el9 2024-04-16T07:09:41.149 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T07:09:41.149 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T07:09:41.151 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:09:41.151 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:09:41.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:09:41.257 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:09:41.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:09:41.353 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:09:41.354 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:09:41.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:09:41.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:09:41.552 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:09:41.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:09:41.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:09:41.737 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:09:41.737 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:09:41.737 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:09:41.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:09:41.981 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:09:41.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:09:42.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:09:42.199 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:09:42.199 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:09:42.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:09:42.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:09:42.408 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:09:42.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:09:42.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:09:42.619 INFO:teuthology.run_tasks:Running task print... 2024-04-16T07:09:42.629 INFO:teuthology.task.print:**** done install task... 2024-04-16T07:09:42.630 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-16T07:09:42.728 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T07:09:42.729 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T07:09:42.729 INFO:tasks.cephadm:Cluster fsid is 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:09:42.730 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-16T07:09:42.730 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-16T07:09:42.730 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi026': '172.21.15.26', 'mon.smithi104': '172.21.15.104'} 2024-04-16T07:09:42.730 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-16T07:09:42.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname $(hostname -s) 2024-04-16T07:09:42.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-04-16T07:09:42.919 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-16T07:09:42.919 DEBUG:teuthology.orchestra.run.smithi026:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.396 INFO:teuthology.orchestra.run.smithi026.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 07:09 /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.396 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.787 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 07:09 /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.787 DEBUG:teuthology.orchestra.run.smithi026:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.852 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T07:09:43.921 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 on all hosts... 2024-04-16T07:09:43.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T07:09:43.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T07:09:45.136 INFO:teuthology.orchestra.run.smithi026.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T07:09:45.164 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T07:10:31.226 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-16T07:10:31.227 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "repo_digests": [ 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-16T07:10:35.416 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:10:35.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph 2024-04-16T07:10:35.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-04-16T07:10:35.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /etc/ceph 2024-04-16T07:10:35.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-04-16T07:10:35.887 INFO:tasks.cephadm:Writing seed config... 2024-04-16T07:10:35.888 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-16T07:10:35.888 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-16T07:10:35.888 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-16T07:10:35.888 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-16T07:10:35.888 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-16T07:10:35.889 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-16T07:10:35.890 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-16T07:10:35.891 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-16T07:10:35.891 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-16T07:10:35.891 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-16T07:10:35.891 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-16T07:10:35.891 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-16T07:10:35.891 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:10:35.891 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-16T07:10:35.950 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-16T07:10:35.950 DEBUG:teuthology.orchestra.run.smithi026:mon.smithi026> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service 2024-04-16T07:10:35.998 INFO:tasks.cephadm:Bootstrapping... 2024-04-16T07:10:35.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 -v bootstrap --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.26 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:10:36.322 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:10:36.322 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', '-v', 'bootstrap', '--fsid', '4c4a53e6-fbc0-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.26', '--skip-admin-label'] 2024-04-16T07:10:36.382 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T07:10:36.383 INFO:teuthology.orchestra.run.smithi026.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-16T07:10:36.383 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying podman|docker is present... 2024-04-16T07:10:36.440 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T07:10:36.440 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying lvm2 is present... 2024-04-16T07:10:36.440 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying time synchronization is in place... 2024-04-16T07:10:36.459 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T07:10:36.459 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T07:10:36.476 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T07:10:36.476 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:10:36.497 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-04-16T07:10:36.518 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-04-16T07:10:36.518 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-04-16T07:10:36.518 INFO:teuthology.orchestra.run.smithi026.stdout:Repeating the final host check... 2024-04-16T07:10:36.584 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T07:10:36.585 INFO:teuthology.orchestra.run.smithi026.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-16T07:10:36.585 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl is present 2024-04-16T07:10:36.585 INFO:teuthology.orchestra.run.smithi026.stdout:lvcreate is present 2024-04-16T07:10:36.606 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T07:10:36.606 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T07:10:36.629 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T07:10:36.629 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:10:36.656 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-04-16T07:10:36.680 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-04-16T07:10:36.681 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-04-16T07:10:36.681 INFO:teuthology.orchestra.run.smithi026.stdout:Host looks OK 2024-04-16T07:10:36.681 INFO:teuthology.orchestra.run.smithi026.stdout:Cluster fsid: 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:36.682 INFO:teuthology.orchestra.run.smithi026.stdout:Acquiring lock 139985129006176 on /run/cephadm/4c4a53e6-fbc0-11ee-bc8f-c7b262605968.lock 2024-04-16T07:10:36.682 INFO:teuthology.orchestra.run.smithi026.stdout:Lock 139985129006176 acquired on /run/cephadm/4c4a53e6-fbc0-11ee-bc8f-c7b262605968.lock 2024-04-16T07:10:36.684 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 3300 ... 2024-04-16T07:10:36.685 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 6789 ... 2024-04-16T07:10:36.685 INFO:teuthology.orchestra.run.smithi026.stdout:Base mon IP(s) is [172.21.15.26:3300, 172.21.15.26:6789], mon addrv is [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-04-16T07:10:36.701 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.26 metric 100 2024-04-16T07:10:36.701 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.26 metric 100 2024-04-16T07:10:36.715 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-16T07:10:36.715 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-16T07:10:36.715 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-16T07:10:36.728 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-16T07:10:36.728 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-16T07:10:36.728 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T07:10:36.728 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-16T07:10:36.729 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-04-16T07:10:36.729 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T07:10:36.731 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-04-16T07:10:36.731 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-04-16T07:10:36.732 INFO:teuthology.orchestra.run.smithi026.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-16T07:10:36.734 INFO:teuthology.orchestra.run.smithi026.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-16T07:10:36.737 INFO:teuthology.orchestra.run.smithi026.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Getting image source signatures 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:0ca96691e690249b227cbeb34b4e7fdd96c9d73da309c97947ab03def7d1129d 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-16T07:10:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying config sha256:b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T07:10:37.477 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-16T07:10:38.708 INFO:teuthology.orchestra.run.smithi026.stdout:ceph: stdout ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T07:10:38.708 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph version: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T07:10:38.708 INFO:teuthology.orchestra.run.smithi026.stdout:Extracting ceph user uid/gid from container image... 2024-04-16T07:10:39.848 INFO:teuthology.orchestra.run.smithi026.stdout:stat: stdout 167 167 2024-04-16T07:10:39.848 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial keys... 2024-04-16T07:10:40.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQBwJB5mQf5rExAAqsYKhkdrMd/2kiynqGE5aQ== 2024-04-16T07:10:42.067 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQBxJB5mgRqwGRAARP61ZRt3mCGTYpO7ZNF3UQ== 2024-04-16T07:10:43.232 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQByJB5mzTDdIBAABS0ZWADtU8whAldlHNmrlw== 2024-04-16T07:10:43.234 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial monmap... 2024-04-16T07:10:44.332 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool for smithi026 [v2:172.21.15.26:3300,v1:172.21.15.26:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = octopus 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: set fsid to 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T07:10:44.333 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:10:44.334 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mon... 2024-04-16T07:10:45.538 INFO:teuthology.orchestra.run.smithi026.stdout:create mon.smithi026 on 2024-04-16T07:10:45.953 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-16T07:10:46.207 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-16T07:10:46.482 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target. 2024-04-16T07:10:46.482 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target. 2024-04-16T07:10:46.805 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026 2024-04-16T07:10:46.805 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service: Unit ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service not loaded. 2024-04-16T07:10:47.097 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target.wants/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service → /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@.service. 2024-04-16T07:10:47.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 podman[28417]: 2024-04-16 07:10:47.543207646 +0000 UTC m=+0.153751545 container create 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:10:47.779 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 podman[28417]: 2024-04-16 07:10:47.779363419 +0000 UTC m=+0.389907298 container init 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-16T07:10:47.882 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T07:10:47.882 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-16T07:10:47.898 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T07:10:47.898 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:10:47.898 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-16T07:10:47.898 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T07:10:47.899 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon to start... 2024-04-16T07:10:47.899 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon... 2024-04-16T07:10:48.047 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 podman[28417]: 2024-04-16 07:10:47.790512332 +0000 UTC m=+0.401056235 container start 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T07:10:48.047 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 bash[28417]: 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 2024-04-16T07:10:48.047 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 systemd[1]: Started Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:10:48.047 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 ceph-mon[28436]: mkfs 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:48.047 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:47 smithi026 ceph-mon[28436]: mon.smithi026 is new leader, mons smithi026 in quorum (ranks 0) 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: mon.smithi026 is new leader, mons smithi026 in quorum (ranks 0) 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: monmap e1: 1 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: fsmap 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: mgrmap e1: no daemons active 2024-04-16T07:10:49.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:48 smithi026 ceph-mon[28436]: from='client.? 172.21.15.26:0/1634317146' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-16T07:10:49.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout cluster: 2024-04-16T07:10:49.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout id: 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:49.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-16T07:10:49.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:49.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout services: 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi026 (age 0.711074s) 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout data: 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pgs: 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.480+0000 7fb8f6400700 1 Processor -- start 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.481+0000 7fb8f6400700 1 -- start start 2024-04-16T07:10:49.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.481+0000 7fb8f6400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.481+0000 7fb8f6400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0105600 con 0x7fb8f0104c20 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.482+0000 7fb8ef400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.482+0000 7fb8ef400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33214/0 (socket says 172.21.15.26:33214) 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.482+0000 7fb8ef400700 1 -- 172.21.15.26:0/600949313 learned_addr learned my addr 172.21.15.26:0/600949313 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.483+0000 7fb8ef400700 1 -- 172.21.15.26:0/600949313 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0105740 con 0x7fb8f0104c20 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.483+0000 7fb8ef400700 1 --2- 172.21.15.26:0/600949313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00098e0 tx=0x7fb8e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd49226743c209d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.483+0000 7fb8ee000700 1 -- 172.21.15.26:0/600949313 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e000b860 con 0x7fb8f0104c20 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.483+0000 7fb8ee000700 1 -- 172.21.15.26:0/600949313 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e000b9c0 con 0x7fb8f0104c20 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.483+0000 7fb8ee000700 1 -- 172.21.15.26:0/600949313 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e000bc90 con 0x7fb8f0104c20 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 -- 172.21.15.26:0/600949313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 msgr2=0x7fb8f0105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 --2- 172.21.15.26:0/600949313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00098e0 tx=0x7fb8e0009250 comp rx=0 tx=0).stop 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 -- 172.21.15.26:0/600949313 shutdown_connections 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 --2- 172.21.15.26:0/600949313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0105030 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:49.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 -- 172.21.15.26:0/600949313 >> 172.21.15.26:0/600949313 conn(0x7fb8f0078350 msgr2=0x7fb8f0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 -- 172.21.15.26:0/600949313 shutdown_connections 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.484+0000 7fb8f6400700 1 -- 172.21.15.26:0/600949313 wait complete. 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.485+0000 7fb8f6400700 1 Processor -- start 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.485+0000 7fb8f6400700 1 -- start start 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.485+0000 7fb8f6400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.485+0000 7fb8f6400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0113470 con 0x7fb8f0104c20 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.486+0000 7fb8ef400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.486+0000 7fb8ef400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33228/0 (socket says 172.21.15.26:33228) 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.486+0000 7fb8ef400700 1 -- 172.21.15.26:0/1634317146 learned_addr learned my addr 172.21.15.26:0/1634317146 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8ef400700 1 -- 172.21.15.26:0/1634317146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e00094f0 con 0x7fb8f0104c20 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8ef400700 1 --2- 172.21.15.26:0/1634317146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0003080 tx=0x7fb8e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e001c5c0 con 0x7fb8f0104c20 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e001c720 con 0x7fb8f0104c20 2024-04-16T07:10:49.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f0113670 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.487+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e001ca30 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.488+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f01141e0 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.488+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb8e001cc60 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.488+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb8e001ac80 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.489+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f005f5f0 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.492+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fb8e001ae40 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.572+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb8f0114cb0 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.573+0000 7fb8e7400700 1 -- 172.21.15.26:0/1634317146 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fb8e001a450 con 0x7fb8f0104c20 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.575+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 msgr2=0x7fb8f0112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.575+0000 7fb8f6400700 1 --2- 172.21.15.26:0/1634317146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0003080 tx=0x7fb8e0016000 comp rx=0 tx=0).stop 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.575+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 shutdown_connections 2024-04-16T07:10:49.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.575+0000 7fb8f6400700 1 --2- 172.21.15.26:0/1634317146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f0104c20 0x7fb8f0112f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:49.179 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.575+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 >> 172.21.15.26:0/1634317146 conn(0x7fb8f0078350 msgr2=0x7fb8f0109760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:49.179 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.576+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 shutdown_connections 2024-04-16T07:10:49.179 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:48.576+0000 7fb8f6400700 1 -- 172.21.15.26:0/1634317146 wait complete. 2024-04-16T07:10:49.179 INFO:teuthology.orchestra.run.smithi026.stdout:mon is available 2024-04-16T07:10:49.179 INFO:teuthology.orchestra.run.smithi026.stdout:Assimilating anything we can from ceph.conf... 2024-04-16T07:10:50.176 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:49 smithi026 ceph-mon[28436]: from='client.? 172.21.15.26:0/1139204611' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T07:10:50.176 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:49 smithi026 ceph-mon[28436]: from='client.? 172.21.15.26:0/1139204611' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:50.509 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.783+0000 7ff8c1600700 1 Processor -- start 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.783+0000 7ff8c1600700 1 -- start start 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.784+0000 7ff8c1600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.784+0000 7ff8c1600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc1030b0 con 0x7ff8bc102760 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.784+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.785+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33244/0 (socket says 172.21.15.26:33244) 2024-04-16T07:10:50.510 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.785+0000 7ff8bbe00700 1 -- 172.21.15.26:0/3719757727 learned_addr learned my addr 172.21.15.26:0/3719757727 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.785+0000 7ff8bbe00700 1 -- 172.21.15.26:0/3719757727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc1031f0 con 0x7ff8bc102760 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.785+0000 7ff8bbe00700 1 --2- 172.21.15.26:0/3719757727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0098e0 tx=0x7ff8ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c911205ce286509e server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8bb400700 1 -- 172.21.15.26:0/3719757727 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ac00b860 con 0x7ff8bc102760 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8bb400700 1 -- 172.21.15.26:0/3719757727 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00b9c0 con 0x7ff8bc102760 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8bb400700 1 -- 172.21.15.26:0/3719757727 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ac00bc90 con 0x7ff8bc102760 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8c1600700 1 -- 172.21.15.26:0/3719757727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 msgr2=0x7ff8bc102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8c1600700 1 --2- 172.21.15.26:0/3719757727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0098e0 tx=0x7ff8ac009250 comp rx=0 tx=0).stop 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8c1600700 1 -- 172.21.15.26:0/3719757727 shutdown_connections 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8c1600700 1 --2- 172.21.15.26:0/3719757727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc102b70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.786+0000 7ff8c1600700 1 -- 172.21.15.26:0/3719757727 >> 172.21.15.26:0/3719757727 conn(0x7ff8bc0fdf40 msgr2=0x7ff8bc100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.787+0000 7ff8c1600700 1 -- 172.21.15.26:0/3719757727 shutdown_connections 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.787+0000 7ff8c1600700 1 -- 172.21.15.26:0/3719757727 wait complete. 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.787+0000 7ff8c1600700 1 Processor -- start 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8c1600700 1 -- start start 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8c1600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:50.511 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8c1600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0791b0 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33252/0 (socket says 172.21.15.26:33252) 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.788+0000 7ff8bbe00700 1 -- 172.21.15.26:0/1139204611 learned_addr learned my addr 172.21.15.26:0/1139204611 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.789+0000 7ff8bbe00700 1 -- 172.21.15.26:0/1139204611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0094f0 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.789+0000 7ff8bbe00700 1 --2- 172.21.15.26:0/1139204611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac003080 tx=0x7ff8ac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.789+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ac01c5c0 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.789+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac01c720 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.789+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc077220 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.790+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ac01ca30 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.790+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc077710 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.790+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff8ac01cc60 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.790+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff8ac01ac80 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.791+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc04bed0 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.794+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7ff8ac01ae40 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.872+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff8bc103990 con 0x7ff8bc102760 2024-04-16T07:10:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.880+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 7 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ff8ac01a450 con 0x7ff8bc102760 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.880+0000 7ff8b8c00700 1 -- 172.21.15.26:0/1139204611 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7ff8ac01b560 con 0x7ff8bc102760 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.881+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 msgr2=0x7ff8bc078c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.881+0000 7ff8c1600700 1 --2- 172.21.15.26:0/1139204611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac003080 tx=0x7ff8ac016000 comp rx=0 tx=0).stop 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.882+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 shutdown_connections 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.882+0000 7ff8c1600700 1 --2- 172.21.15.26:0/1139204611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8bc102760 0x7ff8bc078c70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.882+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 >> 172.21.15.26:0/1139204611 conn(0x7ff8bc0fdf40 msgr2=0x7ff8bc105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.882+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 shutdown_connections 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:49.882+0000 7ff8c1600700 1 -- 172.21.15.26:0/1139204611 wait complete. 2024-04-16T07:10:50.513 INFO:teuthology.orchestra.run.smithi026.stdout:Generating new minimal ceph.conf... 2024-04-16T07:10:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:51 smithi026 ceph-mon[28436]: from='client.? 172.21.15.26:0/3293045706' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:10:51.853 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.065+0000 7fd1ee000700 1 Processor -- start 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.065+0000 7fd1ee000700 1 -- start start 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.065+0000 7fd1ee000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.066+0000 7fd1ee000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e807d420 con 0x7fd1e807ca40 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.066+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.066+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33268/0 (socket says 172.21.15.26:33268) 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.066+0000 7fd1e7400700 1 -- 172.21.15.26:0/544887114 learned_addr learned my addr 172.21.15.26:0/544887114 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.067+0000 7fd1e7400700 1 -- 172.21.15.26:0/544887114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e807d560 con 0x7fd1e807ca40 2024-04-16T07:10:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.067+0000 7fd1e7400700 1 --2- 172.21.15.26:0/544887114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80098e0 tx=0x7fd1d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cd47d7d11ab0c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.068+0000 7fd1e6000700 1 -- 172.21.15.26:0/544887114 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d800b860 con 0x7fd1e807ca40 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.068+0000 7fd1e6000700 1 -- 172.21.15.26:0/544887114 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd1d800be50 con 0x7fd1e807ca40 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.068+0000 7fd1e6000700 1 -- 172.21.15.26:0/544887114 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d8010380 con 0x7fd1e807ca40 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.068+0000 7fd1ee000700 1 -- 172.21.15.26:0/544887114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 msgr2=0x7fd1e807ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.068+0000 7fd1ee000700 1 --2- 172.21.15.26:0/544887114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80098e0 tx=0x7fd1d8009250 comp rx=0 tx=0).stop 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.069+0000 7fd1ee000700 1 -- 172.21.15.26:0/544887114 shutdown_connections 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.069+0000 7fd1ee000700 1 --2- 172.21.15.26:0/544887114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e807ce50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.069+0000 7fd1ee000700 1 -- 172.21.15.26:0/544887114 >> 172.21.15.26:0/544887114 conn(0x7fd1e8078350 msgr2=0x7fd1e807a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.069+0000 7fd1ee000700 1 -- 172.21.15.26:0/544887114 shutdown_connections 2024-04-16T07:10:51.855 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.069+0000 7fd1ee000700 1 -- 172.21.15.26:0/544887114 wait complete. 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.070+0000 7fd1ee000700 1 Processor -- start 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.070+0000 7fd1ee000700 1 -- start start 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.070+0000 7fd1ee000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.070+0000 7fd1ee000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8118120 con 0x7fd1e807ca40 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33284/0 (socket says 172.21.15.26:33284) 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1e7400700 1 -- 172.21.15.26:0/3293045706 learned_addr learned my addr 172.21.15.26:0/3293045706 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1e7400700 1 -- 172.21.15.26:0/3293045706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d80094f0 con 0x7fd1e807ca40 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1e7400700 1 --2- 172.21.15.26:0/3293045706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000c00 tx=0x7fd1d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.071+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d801a8a0 con 0x7fd1e807ca40 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd1d801ae90 con 0x7fd1e807ca40 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e8118320 con 0x7fd1e807ca40 2024-04-16T07:10:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d8019e90 con 0x7fd1e807ca40 2024-04-16T07:10:51.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e8118e90 con 0x7fd1e807ca40 2024-04-16T07:10:51.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd1d8023450 con 0x7fd1e807ca40 2024-04-16T07:10:51.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.072+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd1d8021740 con 0x7fd1e807ca40 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.073+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e805f5f0 con 0x7fd1e807ca40 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.076+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd1d8028060 con 0x7fd1e807ca40 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.152+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd1e81190d0 con 0x7fd1e807ca40 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.153+0000 7fd1df400700 1 -- 172.21.15.26:0/3293045706 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd1d801e070 con 0x7fd1e807ca40 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 msgr2=0x7fd1e8117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:51.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 --2- 172.21.15.26:0/3293045706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000c00 tx=0x7fd1d8009250 comp rx=0 tx=0).stop 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 shutdown_connections 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 --2- 172.21.15.26:0/3293045706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1e807ca40 0x7fd1e8117be0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 >> 172.21.15.26:0/3293045706 conn(0x7fd1e8078350 msgr2=0x7fd1e8079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.154+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 shutdown_connections 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:51.155+0000 7fd1ee000700 1 -- 172.21.15.26:0/3293045706 wait complete. 2024-04-16T07:10:51.859 INFO:teuthology.orchestra.run.smithi026.stdout:Restarting the monitor... 2024-04-16T07:10:52.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:51 smithi026 systemd[1]: Stopping Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:10:52.745 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:52 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[28432]: 2024-04-16T07:10:52.363+0000 7f6b4fe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi026 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:10:52.745 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:52 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[28432]: 2024-04-16T07:10:52.363+0000 7f6b4fe00700 -1 mon.smithi026@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T07:10:52.745 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:52 smithi026 podman[28680]: 2024-04-16 07:10:52.436630374 +0000 UTC m=+0.183501183 container died 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:10:53.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:52 smithi026 podman[28680]: 2024-04-16 07:10:52.857472632 +0000 UTC m=+0.604343492 container cleanup 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:10:53.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:52 smithi026 bash[28680]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026 2024-04-16T07:10:53.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 podman[28694]: 2024-04-16 07:10:53.133667598 +0000 UTC m=+0.675392126 container remove 0de2b538b679832ea5846323c229097ec84dd14a54e22a54f40785317dee9905 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-16T07:10:53.981 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service: Deactivated successfully. 2024-04-16T07:10:53.981 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 systemd[1]: Stopped Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:10:53.981 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service: Consumed 1.934s CPU time. 2024-04-16T07:10:53.981 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 systemd[1]: Starting Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:10:54.246 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:53 smithi026 podman[28785]: 2024-04-16 07:10:53.887571373 +0000 UTC m=+0.034091993 image pull b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T07:10:54.246 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 podman[28785]: 2024-04-16 07:10:54.010163192 +0000 UTC m=+0.156683850 container create 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T07:10:54.246 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 podman[28785]: 2024-04-16 07:10:54.246851453 +0000 UTC m=+0.393372118 container init 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-16T07:10:54.345 INFO:teuthology.orchestra.run.smithi026.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-16T07:10:54.502 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 podman[28785]: 2024-04-16 07:10:54.257643988 +0000 UTC m=+0.404164619 container start 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-mon, pid 2 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: pidfile_write: ignore empty --pid-file 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: load: jerasure load: lrc 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: RocksDB version: 6.15.5 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Compile date Apr 12 2024 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: DB SUMMARY 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: DB Session ID: Q6734BKM3WVP7AP5RI52 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: CURRENT file: CURRENT 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi026/store.db dir, Total Num: 1, files: 000008.sst 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi026/store.db: 000010.log size: 98290 ; 2024-04-16T07:10:54.503 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.error_if_exists: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.create_if_missing: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.paranoid_checks: 1 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.env: 0x56176f695b00 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.fs: Posix File System 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.info_log: 0x56177bb85c00 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.statistics: (nil) 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.use_fsync: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_log_file_size: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_fallocate: 1 2024-04-16T07:10:54.504 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.use_direct_reads: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.db_log_dir: 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi026/store.db 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T07:10:54.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.write_buffer_manager: 0x56177c8faa20 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T07:10:54.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.unordered_write: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.row_cache: None 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.wal_filter: None 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.preserve_deletes: 0 2024-04-16T07:10:54.507 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.two_write_queues: 0 2024-04-16T07:10:54.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T07:10:54.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.atomic_flush: 0 2024-04-16T07:10:54.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T07:10:54.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T07:10:54.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.log_readahead_size: 0 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_background_jobs: 2 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_background_compactions: -1 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_subcompactions: 1 2024-04-16T07:10:54.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_open_files: -1 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T07:10:54.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_background_flushes: -1 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Compression algorithms supported: 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kZSTD supported: 0 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kXpressCompression supported: 0 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kZlibCompression supported: 1 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kSnappyCompression supported: 1 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kLZ4Compression supported: 1 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: kBZip2Compression supported: 0 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000009 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T07:10:54.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.merge_operator: 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_filter: None 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56177bbdaf58) 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_type: 0 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_index_type: 0 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_shortening: 1 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: hash_index_allow_collision: 1 2024-04-16T07:10:54.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: checksum: 1 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: no_block_cache: 0 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache: 0x56177bc0d090 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_options: 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: capacity : 536870912 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: num_shard_bits : 4 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: strict_capacity_limit : 0 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_compressed: (nil) 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: persistent_cache: (nil) 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size: 4096 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size_deviation: 10 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_restart_interval: 16 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_block_restart_interval: 1 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: metadata_block_size: 4096 2024-04-16T07:10:54.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: partition_filters: 0 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: use_delta_encoding: 1 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: whole_key_filtering: 1 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: verify_compression: 0 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: format_version: 4 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_index_compression: 1 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_align: 0 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression: NoCompression 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T07:10:54.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.num_levels: 7 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T07:10:54.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T07:10:54.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.arena_block_size: 4194304 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T07:10:54.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.table_properties_collectors: 2024-04-16T07:10:54.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.inplace_update_support: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.bloom_locality: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.max_successive_merges: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.ttl: 2592000 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.enable_blob_files: false 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.min_blob_size: 0 2024-04-16T07:10:54.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251454308631, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251454310601, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93552, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91636, "index_size": 276, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84172, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713251454, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ec0f0823-6219-414d-9944-e406e9cc67c4", "db_session_id": "Q6734BKM3WVP7AP5RI52"}} 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251454311360, "job": 1, "event": "recovery_finished"} 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi026/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56177bba1180 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: DB pointer 0x56177bc76000 2024-04-16T07:10:54.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** DB Stats ** 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: L0 2/0 93.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Sum 2/0 93.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative compaction: 0.00 GB write, 12.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval compaction: 0.00 GB write, 12.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:10:54.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: L0 2/0 93.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Sum 2/0 93.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 62.8 0.00 0.00 1 0.001 0 0 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T07:10:54.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative compaction: 0.00 GB write, 12.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: starting mon.smithi026 rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi026 fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???) e1 preinit fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).mds e1 new map 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).mds e1 print_map 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: e1 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:10:54.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: legacy client fscid: -1 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: No filesystems configured 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 bash[28785]: 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 systemd[1]: Started Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mon.smithi026 is new leader, mons smithi026 in quorum (ranks 0) 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: monmap e1: 1 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: fsmap 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T07:10:54.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:54 smithi026 ceph-mon[28805]: mgrmap e1: no daemons active 2024-04-16T07:10:55.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.038+0000 7fa4b5400700 1 Processor -- start 2024-04-16T07:10:55.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.038+0000 7fa4b5400700 1 -- start start 2024-04-16T07:10:55.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.038+0000 7fa4b5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:55.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.038+0000 7fa4b5400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01031e0 con 0x7fa4b0102890 2024-04-16T07:10:55.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.039+0000 7fa4ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:55.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.039+0000 7fa4ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33294/0 (socket says 172.21.15.26:33294) 2024-04-16T07:10:55.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.039+0000 7fa4ae800700 1 -- 172.21.15.26:0/3555314958 learned_addr learned my addr 172.21.15.26:0/3555314958 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:55.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.040+0000 7fa4ae800700 1 -- 172.21.15.26:0/3555314958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0103320 con 0x7fa4b0102890 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.040+0000 7fa4ae800700 1 --2- 172.21.15.26:0/3555314958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00098e0 tx=0x7fa4a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a16c22ef008b9a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.041+0000 7fa4ad400700 1 -- 172.21.15.26:0/3555314958 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a000b860 con 0x7fa4b0102890 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.041+0000 7fa4ad400700 1 -- 172.21.15.26:0/3555314958 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa4a000be50 con 0x7fa4b0102890 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.041+0000 7fa4ad400700 1 -- 172.21.15.26:0/3555314958 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a0010380 con 0x7fa4b0102890 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.041+0000 7fa4b5400700 1 -- 172.21.15.26:0/3555314958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 msgr2=0x7fa4b0102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.041+0000 7fa4b5400700 1 --2- 172.21.15.26:0/3555314958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00098e0 tx=0x7fa4a0009250 comp rx=0 tx=0).stop 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.042+0000 7fa4b5400700 1 -- 172.21.15.26:0/3555314958 shutdown_connections 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.042+0000 7fa4b5400700 1 --2- 172.21.15.26:0/3555314958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b0102ca0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.042+0000 7fa4b5400700 1 -- 172.21.15.26:0/3555314958 >> 172.21.15.26:0/3555314958 conn(0x7fa4b00fe090 msgr2=0x7fa4b01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.042+0000 7fa4b5400700 1 -- 172.21.15.26:0/3555314958 shutdown_connections 2024-04-16T07:10:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.042+0000 7fa4b5400700 1 -- 172.21.15.26:0/3555314958 wait complete. 2024-04-16T07:10:55.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.043+0000 7fa4b5400700 1 Processor -- start 2024-04-16T07:10:55.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.043+0000 7fa4b5400700 1 -- start start 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4b5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4b5400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0113d00 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33298/0 (socket says 172.21.15.26:33298) 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4ae800700 1 -- 172.21.15.26:0/1644836150 learned_addr learned my addr 172.21.15.26:0/1644836150 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.044+0000 7fa4ae800700 1 -- 172.21.15.26:0/1644836150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a00094f0 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.045+0000 7fa4ae800700 1 --2- 172.21.15.26:0/1644836150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0000c00 tx=0x7fa4a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.045+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a001a870 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.045+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa4a001ae60 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.045+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0113f00 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.045+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a00193c0 con 0x7fa4b0102890 2024-04-16T07:10:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.046+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa4a0019520 con 0x7fa4b0102890 2024-04-16T07:10:55.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.046+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0114a70 con 0x7fa4b0102890 2024-04-16T07:10:55.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.046+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa4a00227b0 con 0x7fa4b0102890 2024-04-16T07:10:55.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.047+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b005f5f0 con 0x7fa4b0102890 2024-04-16T07:10:55.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.050+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fa4a001a9d0 con 0x7fa4b0102890 2024-04-16T07:10:55.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.128+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa4b0104670 con 0x7fa4b0102890 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.131+0000 7fa4a6a00700 1 -- 172.21.15.26:0/1644836150 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa4a001ab70 con 0x7fa4b0102890 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.132+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 msgr2=0x7fa4b01137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.132+0000 7fa4b5400700 1 --2- 172.21.15.26:0/1644836150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0000c00 tx=0x7fa4a0009250 comp rx=0 tx=0).stop 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.132+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 shutdown_connections 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.132+0000 7fa4b5400700 1 --2- 172.21.15.26:0/1644836150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4b0102890 0x7fa4b01137c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.132+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 >> 172.21.15.26:0/1644836150 conn(0x7fa4b00fe090 msgr2=0x7fa4b00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.133+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 shutdown_connections 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:55.133+0000 7fa4b5400700 1 -- 172.21.15.26:0/1644836150 wait complete. 2024-04-16T07:10:55.767 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-16T07:10:55.768 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:10:55.768 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mgr... 2024-04-16T07:10:55.769 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 9283 ... 2024-04-16T07:10:56.097 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mgr.smithi026.tawexg 2024-04-16T07:10:56.097 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mgr.smithi026.tawexg.service: Unit ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mgr.smithi026.tawexg.service not loaded. 2024-04-16T07:10:56.367 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968.target.wants/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mgr.smithi026.tawexg.service → /etc/systemd/system/ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@.service. 2024-04-16T07:10:56.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:56 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1644836150' entity='client.admin' 2024-04-16T07:10:57.138 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T07:10:57.138 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-16T07:10:57.155 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T07:10:57.155 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:10:57.155 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-16T07:10:57.156 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T07:10:57.185 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T07:10:57.185 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-16T07:10:57.209 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T07:10:57.210 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:10:57.210 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-16T07:10:57.210 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-16T07:10:57.211 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr to start... 2024-04-16T07:10:57.211 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr... 2024-04-16T07:10:58.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:10:57 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4046619704' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T07:11:02.834 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:11:02.834 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:11:02.834 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "4c4a53e6-fbc0-11ee-bc8f-c7b262605968", 2024-04-16T07:11:02.834 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-16T07:11:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T07:11:02.837 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T07:11:02.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T07:11:02.845 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T07:11:02.845 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T07:11:02.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:02.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T07:10:47.865368+0000", 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.815+0000 7effbf200700 1 Processor -- start 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.815+0000 7effbf200700 1 -- start start 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.816+0000 7effbf200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.816+0000 7effbf200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0078080 con 0x7effc0077c70 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.816+0000 7effbde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.816+0000 7effbde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33312/0 (socket says 172.21.15.26:33312) 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.816+0000 7effbde00700 1 -- 172.21.15.26:0/1660581427 learned_addr learned my addr 172.21.15.26:0/1660581427 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.817+0000 7effbde00700 1 -- 172.21.15.26:0/1660581427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00766f0 con 0x7effc0077c70 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.817+0000 7effbde00700 1 --2- 172.21.15.26:0/1660581427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7effb00098e0 tx=0x7effb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2224bb16aa92e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbca00700 1 -- 172.21.15.26:0/1660581427 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb000b860 con 0x7effc0077c70 2024-04-16T07:11:02.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbca00700 1 -- 172.21.15.26:0/1660581427 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7effb000be50 con 0x7effc0077c70 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbca00700 1 -- 172.21.15.26:0/1660581427 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb0010380 con 0x7effc0077c70 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbf200700 1 -- 172.21.15.26:0/1660581427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 msgr2=0x7effc00761b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbf200700 1 --2- 172.21.15.26:0/1660581427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7effb00098e0 tx=0x7effb0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbf200700 1 -- 172.21.15.26:0/1660581427 shutdown_connections 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbf200700 1 --2- 172.21.15.26:0/1660581427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0077c70 0x7effc00761b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.818+0000 7effbf200700 1 -- 172.21.15.26:0/1660581427 >> 172.21.15.26:0/1660581427 conn(0x7effc00fde30 msgr2=0x7effc0100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.819+0000 7effbf200700 1 -- 172.21.15.26:0/1660581427 shutdown_connections 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.819+0000 7effbf200700 1 -- 172.21.15.26:0/1660581427 wait complete. 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbf200700 1 Processor -- start 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbf200700 1 -- start start 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbf200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbf200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0118530 con 0x7effc0117be0 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33322/0 (socket says 172.21.15.26:33322) 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.820+0000 7effbde00700 1 -- 172.21.15.26:0/4046619704 learned_addr learned my addr 172.21.15.26:0/4046619704 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:02.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effbde00700 1 -- 172.21.15.26:0/4046619704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb00094f0 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effbde00700 1 --2- 172.21.15.26:0/4046619704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7effc00779b0 tx=0x7effb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb001a7a0 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7effb001ad90 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb0024c70 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.821+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0118730 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.822+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0119110 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.822+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7effb002d3f0 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.822+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7effb0019710 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.823+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc005f5f0 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.826+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7effb0029070 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.905+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7effc0104670 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.906+0000 7effb6000700 1 -- 172.21.15.26:0/4046619704 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7effb0024580 con 0x7effc0117be0 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 msgr2=0x7effc0117ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 --2- 172.21.15.26:0/4046619704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7effc00779b0 tx=0x7effb0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:02.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 shutdown_connections 2024-04-16T07:11:02.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 --2- 172.21.15.26:0/4046619704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effc0117be0 0x7effc0117ff0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:02.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 >> 172.21.15.26:0/4046619704 conn(0x7effc00fde30 msgr2=0x7effc010edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:02.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 shutdown_connections 2024-04-16T07:11:02.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:10:57.908+0000 7effbf200700 1 -- 172.21.15.26:0/4046619704 wait complete. 2024-04-16T07:11:02.851 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (1/15)... 2024-04-16T07:11:06.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: Activating manager daemon smithi026.tawexg 2024-04-16T07:11:06.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: mgrmap e2: smithi026.tawexg(active, starting, since 0.0109538s) 2024-04-16T07:11:06.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:06 smithi026 ceph-mon[28805]: from='mgr.14100 172.21.15.26:0/639113393' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:08.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:08 smithi026 ceph-mon[28805]: mgrmap e3: smithi026.tawexg(active, since 1.02394s) 2024-04-16T07:11:08.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:08 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1457689203' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "4c4a53e6-fbc0-11ee-bc8f-c7b262605968", 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T07:11:08.434 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T07:11:08.435 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T07:11:08.436 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T07:11:08.436 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T07:11:08.444 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T07:11:08.444 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T07:11:08.445 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.446 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T07:10:47.865368+0000", 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.520+0000 7f622b400700 1 Processor -- start 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.520+0000 7f622b400700 1 -- start start 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.520+0000 7f622b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.520+0000 7f622b400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622407d420 con 0x7f622407ca40 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.521+0000 7f6228a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.521+0000 7f6228a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59028/0 (socket says 172.21.15.26:59028) 2024-04-16T07:11:08.447 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.521+0000 7f6228a00700 1 -- 172.21.15.26:0/1464402028 learned_addr learned my addr 172.21.15.26:0/1464402028 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.521+0000 7f6228a00700 1 -- 172.21.15.26:0/1464402028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622407d560 con 0x7f622407ca40 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.522+0000 7f6228a00700 1 --2- 172.21.15.26:0/1464402028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62180098e0 tx=0x7f6218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955e69dafd780f10 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.522+0000 7f6223400700 1 -- 172.21.15.26:0/1464402028 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621800b860 con 0x7f622407ca40 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.522+0000 7f6223400700 1 -- 172.21.15.26:0/1464402028 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f621800be50 con 0x7f622407ca40 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.522+0000 7f6223400700 1 -- 172.21.15.26:0/1464402028 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6218010380 con 0x7f622407ca40 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 -- 172.21.15.26:0/1464402028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 msgr2=0x7f622407ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 --2- 172.21.15.26:0/1464402028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62180098e0 tx=0x7f6218009250 comp rx=0 tx=0).stop 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 -- 172.21.15.26:0/1464402028 shutdown_connections 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 --2- 172.21.15.26:0/1464402028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f622407ce50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 -- 172.21.15.26:0/1464402028 >> 172.21.15.26:0/1464402028 conn(0x7f6224078350 msgr2=0x7f622407a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 -- 172.21.15.26:0/1464402028 shutdown_connections 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.523+0000 7f622b400700 1 -- 172.21.15.26:0/1464402028 wait complete. 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.524+0000 7f622b400700 1 Processor -- start 2024-04-16T07:11:08.448 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.524+0000 7f622b400700 1 -- start start 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.524+0000 7f622b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.524+0000 7f622b400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224118120 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.524+0000 7f6228a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.525+0000 7f6228a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59036/0 (socket says 172.21.15.26:59036) 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.525+0000 7f6228a00700 1 -- 172.21.15.26:0/1457689203 learned_addr learned my addr 172.21.15.26:0/1457689203 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.525+0000 7f6228a00700 1 -- 172.21.15.26:0/1457689203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180094f0 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.525+0000 7f6228a00700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6218006fd0 tx=0x7f6218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.525+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6218010800 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.526+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f6218010df0 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.526+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6224118320 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.526+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621801ad80 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.526+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6224118e90 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.527+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 3) v1 ==== 43259+0+0 (secure 0 0 0) 0x7f621801a660 con 0x7f622407ca40 2024-04-16T07:11:08.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.527+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622405f5f0 con 0x7f622407ca40 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.527+0000 7f6220c00700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 0x7f620c038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.527+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6218049db0 con 0x7f622407ca40 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.528+0000 7f6223e00700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 0x7f620c038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.528+0000 7f6223e00700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 0x7f620c038b40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6210009990 tx=0x7f6210006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.534+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f62180224e0 con 0x7f622407ca40 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.828+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f622404bed0 con 0x7f622407ca40 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.829+0000 7f6220c00700 1 -- 172.21.15.26:0/1457689203 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f6218022300 con 0x7f622407ca40 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 msgr2=0x7f620c038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 0x7f620c038b40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6210009990 tx=0x7f6210006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 msgr2=0x7f6224117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6218006fd0 tx=0x7f6218009250 comp rx=0 tx=0).stop 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 shutdown_connections 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f620c036690 0x7f620c038b40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 --2- 172.21.15.26:0/1457689203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f622407ca40 0x7f6224117be0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:08.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 >> 172.21.15.26:0/1457689203 conn(0x7f6224078350 msgr2=0x7f6224079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:08.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 shutdown_connections 2024-04-16T07:11:08.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:07.832+0000 7f622b400700 1 -- 172.21.15.26:0/1457689203 wait complete. 2024-04-16T07:11:08.451 INFO:teuthology.orchestra.run.smithi026.stdout:mgr is available 2024-04-16T07:11:09.278 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:09 smithi026 ceph-mon[28805]: mgrmap e4: smithi026.tawexg(active, since 2s) 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T07:11:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T07:11:09.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.025+0000 7fab53000700 1 Processor -- start 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.026+0000 7fab53000700 1 -- start start 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.026+0000 7fab53000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:09.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.026+0000 7fab53000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c1031c0 con 0x7fab4c102870 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.027+0000 7fab4be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.027+0000 7fab4be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60444/0 (socket says 172.21.15.26:60444) 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.027+0000 7fab4be00700 1 -- 172.21.15.26:0/1782308730 learned_addr learned my addr 172.21.15.26:0/1782308730 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.027+0000 7fab4be00700 1 -- 172.21.15.26:0/1782308730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c103300 con 0x7fab4c102870 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.028+0000 7fab4be00700 1 --2- 172.21.15.26:0/1782308730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0098e0 tx=0x7fab3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dcbab5635ba3c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.028+0000 7fab4aa00700 1 -- 172.21.15.26:0/1782308730 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c00b860 con 0x7fab4c102870 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.028+0000 7fab4aa00700 1 -- 172.21.15.26:0/1782308730 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fab3c00be50 con 0x7fab4c102870 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.028+0000 7fab4aa00700 1 -- 172.21.15.26:0/1782308730 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c010380 con 0x7fab4c102870 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 -- 172.21.15.26:0/1782308730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 msgr2=0x7fab4c102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 --2- 172.21.15.26:0/1782308730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0098e0 tx=0x7fab3c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 -- 172.21.15.26:0/1782308730 shutdown_connections 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 --2- 172.21.15.26:0/1782308730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c102c80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 -- 172.21.15.26:0/1782308730 >> 172.21.15.26:0/1782308730 conn(0x7fab4c0fe010 msgr2=0x7fab4c100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 -- 172.21.15.26:0/1782308730 shutdown_connections 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.029+0000 7fab53000700 1 -- 172.21.15.26:0/1782308730 wait complete. 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.030+0000 7fab53000700 1 Processor -- start 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.030+0000 7fab53000700 1 -- start start 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.030+0000 7fab53000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.030+0000 7fab53000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c111b20 con 0x7fab4c102870 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.031+0000 7fab4be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.031+0000 7fab4be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60448/0 (socket says 172.21.15.26:60448) 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.031+0000 7fab4be00700 1 -- 172.21.15.26:0/1297500980 learned_addr learned my addr 172.21.15.26:0/1297500980 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:09.947 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.031+0000 7fab4be00700 1 -- 172.21.15.26:0/1297500980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c0094f0 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab4be00700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000c00 tx=0x7fab3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c01a870 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fab3c01ae60 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c0193c0 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c111d20 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.032+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c112890 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.033+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7fab3c022770 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.033+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c05f5f0 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.034+0000 7fab43e00700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 0x7fab30038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.034+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fab3c04a420 con 0x7fab4c102870 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.035+0000 7fab4b400700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 0x7fab30038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:09.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.036+0000 7fab4b400700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 0x7fab30038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab38009990 tx=0x7fab38006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.040+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fab3c010c00 con 0x7fab4c102870 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.256+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fab4c04bed0 con 0x7fab4c102870 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.258+0000 7fab43e00700 1 -- 172.21.15.26:0/1297500980 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fab3c00ee60 con 0x7fab4c102870 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 msgr2=0x7fab30038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 0x7fab30038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab38009990 tx=0x7fab38006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 msgr2=0x7fab4c1115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000c00 tx=0x7fab3c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 shutdown_connections 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fab30036850 0x7fab30038d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 --2- 172.21.15.26:0/1297500980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab4c102870 0x7fab4c1115e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.261+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 >> 172.21.15.26:0/1297500980 conn(0x7fab4c0fe010 msgr2=0x7fab4c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.262+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 shutdown_connections 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:09.262+0000 7fab53000700 1 -- 172.21.15.26:0/1297500980 wait complete. 2024-04-16T07:11:09.949 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling cephadm module... 2024-04-16T07:11:10.246 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:10 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1297500980' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T07:11:11.103 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:11 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/738839738' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-16T07:11:12.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.565+0000 7fa9b0200700 1 Processor -- start 2024-04-16T07:11:12.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.565+0000 7fa9b0200700 1 -- start start 2024-04-16T07:11:12.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.566+0000 7fa9b0200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.566+0000 7fa9b0200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8077f40 con 0x7fa9a8077b30 2024-04-16T07:11:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.566+0000 7fa9ad800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.566+0000 7fa9ad800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60450/0 (socket says 172.21.15.26:60450) 2024-04-16T07:11:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.566+0000 7fa9ad800700 1 -- 172.21.15.26:0/3990399370 learned_addr learned my addr 172.21.15.26:0/3990399370 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.567+0000 7fa9ad800700 1 -- 172.21.15.26:0/3990399370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8076670 con 0x7fa9a8077b30 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.567+0000 7fa9ad800700 1 --2- 172.21.15.26:0/3990399370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098e0 tx=0x7fa99c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30137cabde9746d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.567+0000 7fa9a7e00700 1 -- 172.21.15.26:0/3990399370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c00b860 con 0x7fa9a8077b30 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.567+0000 7fa9a7e00700 1 -- 172.21.15.26:0/3990399370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa99c00be50 con 0x7fa9a8077b30 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.567+0000 7fa9a7e00700 1 -- 172.21.15.26:0/3990399370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c010380 con 0x7fa9a8077b30 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 -- 172.21.15.26:0/3990399370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 msgr2=0x7fa9a8076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 --2- 172.21.15.26:0/3990399370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098e0 tx=0x7fa99c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 -- 172.21.15.26:0/3990399370 shutdown_connections 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 --2- 172.21.15.26:0/3990399370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8076130 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 -- 172.21.15.26:0/3990399370 >> 172.21.15.26:0/3990399370 conn(0x7fa9a80fe110 msgr2=0x7fa9a8100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 -- 172.21.15.26:0/3990399370 shutdown_connections 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.568+0000 7fa9b0200700 1 -- 172.21.15.26:0/3990399370 wait complete. 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.569+0000 7fa9b0200700 1 Processor -- start 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.569+0000 7fa9b0200700 1 -- start start 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.569+0000 7fa9b0200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:12.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.569+0000 7fa9b0200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8113d90 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.570+0000 7fa9ad800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.570+0000 7fa9ad800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60458/0 (socket says 172.21.15.26:60458) 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.570+0000 7fa9ad800700 1 -- 172.21.15.26:0/738839738 learned_addr learned my addr 172.21.15.26:0/738839738 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.570+0000 7fa9ad800700 1 -- 172.21.15.26:0/738839738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0094f0 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.570+0000 7fa9ad800700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa99c006fd0 tx=0x7fa99c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.571+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c01a7d0 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.571+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa99c01adc0 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.571+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c019ea0 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.571+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a8113f90 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.571+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a8114b00 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.572+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a805f5f0 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.572+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7fa99c019730 con 0x7fa9a8077b30 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.573+0000 7fa9a5600700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 0x7fa990038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:12.122 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.573+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa99c049ff0 con 0x7fa9a8077b30 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.573+0000 7fa9ace00700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 0x7fa990038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.574+0000 7fa9ace00700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 0x7fa990038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa994009990 tx=0x7fa994006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.578+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa99c022b60 con 0x7fa9a8077b30 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:10.845+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa9a804bed0 con 0x7fa9a8077b30 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.103+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa99c01fd40 con 0x7fa9a8077b30 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.106+0000 7fa9a5600700 1 -- 172.21.15.26:0/738839738 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa99c04e450 con 0x7fa9a8077b30 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 msgr2=0x7fa990038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 0x7fa990038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa994009990 tx=0x7fa994006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 msgr2=0x7fa9a8113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa99c006fd0 tx=0x7fa99c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 shutdown_connections 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7fa990036850 0x7fa990038d00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 --2- 172.21.15.26:0/738839738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9a8077b30 0x7fa9a8113850 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.108+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 >> 172.21.15.26:0/738839738 conn(0x7fa9a80fe110 msgr2=0x7fa9a80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:12.124 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.109+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 shutdown_connections 2024-04-16T07:11:12.124 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:11.109+0000 7fa9b0200700 1 -- 172.21.15.26:0/738839738 wait complete. 2024-04-16T07:11:12.406 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:12 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/738839738' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-16T07:11:12.406 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:12 smithi026 ceph-mon[28805]: mgrmap e5: smithi026.tawexg(active, since 5s) 2024-04-16T07:11:13.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:13 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3409877939' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "smithi026.tawexg", 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T07:11:13.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.697+0000 7f8e59e00700 1 Processor -- start 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.697+0000 7f8e59e00700 1 -- start start 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.697+0000 7f8e59e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.697+0000 7f8e59e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e54103250 con 0x7f8e54102900 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.698+0000 7f8e52a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.698+0000 7f8e52a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60480/0 (socket says 172.21.15.26:60480) 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.698+0000 7f8e52a00700 1 -- 172.21.15.26:0/110102970 learned_addr learned my addr 172.21.15.26:0/110102970 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.698+0000 7f8e52a00700 1 -- 172.21.15.26:0/110102970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e54103390 con 0x7f8e54102900 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.698+0000 7f8e52a00700 1 --2- 172.21.15.26:0/110102970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8e440098e0 tx=0x7f8e44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d277ce70254e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:13.764 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.699+0000 7f8e51600700 1 -- 172.21.15.26:0/110102970 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4400b860 con 0x7f8e54102900 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.699+0000 7f8e51600700 1 -- 172.21.15.26:0/110102970 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8e4400be50 con 0x7f8e54102900 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.699+0000 7f8e51600700 1 -- 172.21.15.26:0/110102970 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e44010380 con 0x7f8e54102900 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.699+0000 7f8e59e00700 1 -- 172.21.15.26:0/110102970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 msgr2=0x7f8e54102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.699+0000 7f8e59e00700 1 --2- 172.21.15.26:0/110102970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8e440098e0 tx=0x7f8e44009250 comp rx=0 tx=0).stop 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.700+0000 7f8e59e00700 1 -- 172.21.15.26:0/110102970 shutdown_connections 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.700+0000 7f8e59e00700 1 --2- 172.21.15.26:0/110102970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54102d10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.700+0000 7f8e59e00700 1 -- 172.21.15.26:0/110102970 >> 172.21.15.26:0/110102970 conn(0x7f8e540fe0a0 msgr2=0x7f8e541004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.700+0000 7f8e59e00700 1 -- 172.21.15.26:0/110102970 shutdown_connections 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.700+0000 7f8e59e00700 1 -- 172.21.15.26:0/110102970 wait complete. 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.701+0000 7f8e59e00700 1 Processor -- start 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.701+0000 7f8e59e00700 1 -- start start 2024-04-16T07:11:13.765 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e59e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e59e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e54111b50 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e52a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e52a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60488/0 (socket says 172.21.15.26:60488) 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e52a00700 1 -- 172.21.15.26:0/3409877939 learned_addr learned my addr 172.21.15.26:0/3409877939 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.702+0000 7f8e52a00700 1 -- 172.21.15.26:0/3409877939 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440094f0 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e52a00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4401a8a0 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8e4401ae90 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e44019e70 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e54111d50 con 0x7f8e54102900 2024-04-16T07:11:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.703+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e541128c0 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.704+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5405f5f0 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.704+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f8e44019730 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.705+0000 7f8e4aa00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 0x7f8e38038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.705+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8e44049ff0 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.705+0000 7f8e52000700 1 -- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 msgr2=0x7f8e38038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.705+0000 7f8e52000700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 0x7f8e38038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.710+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8e4407c250 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.906+0000 7f8e52000700 1 -- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 msgr2=0x7f8e38038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.906+0000 7f8e52000700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 0x7f8e38038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.980+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8e54112c10 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.981+0000 7f8e4aa00700 1 -- 172.21.15.26:0/3409877939 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f8e4401e020 con 0x7f8e54102900 2024-04-16T07:11:13.767 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.983+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 msgr2=0x7f8e38038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.983+0000 7f8e59e00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 0x7f8e38038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.983+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 msgr2=0x7f8e54111610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.983+0000 7f8e59e00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44009250 comp rx=0 tx=0).stop 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 shutdown_connections 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f8e380368a0 0x7f8e38038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 --2- 172.21.15.26:0/3409877939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e54102900 0x7f8e54111610 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 >> 172.21.15.26:0/3409877939 conn(0x7f8e540fe0a0 msgr2=0x7f8e540feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 shutdown_connections 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:12.984+0000 7f8e59e00700 1 -- 172.21.15.26:0/3409877939 wait complete. 2024-04-16T07:11:13.768 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-04-16T07:11:13.769 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 5... 2024-04-16T07:11:20.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: Active manager daemon smithi026.tawexg restarted 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: Activating manager daemon smithi026.tawexg 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: osdmap e2: 0 total, 0 up, 0 in 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: mgrmap e6: smithi026.tawexg(active, starting, since 0.00988541s) 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:19 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:21.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: Found migration_current of "None". Setting to last migration. 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:11:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:20 smithi026 ceph-mon[28805]: mgrmap e7: smithi026.tawexg(active, since 1.01532s) 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.340+0000 7f5a4f800700 1 Processor -- start 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.340+0000 7f5a4f800700 1 -- start start 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.341+0000 7f5a4f800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.341+0000 7f5a4f800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48077f90 con 0x7f5a48077b80 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.341+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:21.741 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.341+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60500/0 (socket says 172.21.15.26:60500) 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.341+0000 7f5a4ce00700 1 -- 172.21.15.26:0/2933591722 learned_addr learned my addr 172.21.15.26:0/2933591722 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.342+0000 7f5a4ce00700 1 -- 172.21.15.26:0/2933591722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48076670 con 0x7f5a48077b80 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.342+0000 7f5a4ce00700 1 --2- 172.21.15.26:0/2933591722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0098e0 tx=0x7f5a3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37e896382e55cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.342+0000 7f5a47400700 1 -- 172.21.15.26:0/2933591722 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3c00b860 con 0x7f5a48077b80 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.342+0000 7f5a47400700 1 -- 172.21.15.26:0/2933591722 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f5a3c00be50 con 0x7f5a48077b80 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.342+0000 7f5a47400700 1 -- 172.21.15.26:0/2933591722 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3c010380 con 0x7f5a48077b80 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 -- 172.21.15.26:0/2933591722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 msgr2=0x7f5a48076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 --2- 172.21.15.26:0/2933591722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0098e0 tx=0x7f5a3c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 -- 172.21.15.26:0/2933591722 shutdown_connections 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 --2- 172.21.15.26:0/2933591722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48076130 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 -- 172.21.15.26:0/2933591722 >> 172.21.15.26:0/2933591722 conn(0x7f5a480fe0a0 msgr2=0x7f5a481004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 -- 172.21.15.26:0/2933591722 shutdown_connections 2024-04-16T07:11:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.343+0000 7f5a4f800700 1 -- 172.21.15.26:0/2933591722 wait complete. 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.344+0000 7f5a4f800700 1 Processor -- start 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.344+0000 7f5a4f800700 1 -- start start 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.344+0000 7f5a4f800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.344+0000 7f5a4f800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48113da0 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.345+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.345+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60502/0 (socket says 172.21.15.26:60502) 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.345+0000 7f5a4ce00700 1 -- 172.21.15.26:0/3820978370 learned_addr learned my addr 172.21.15.26:0/3820978370 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.345+0000 7f5a4ce00700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0094f0 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.345+0000 7f5a4ce00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c006fd0 tx=0x7f5a3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.346+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3c01a7d0 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.346+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f5a3c01adc0 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.346+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3c021360 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.346+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a48113fa0 con 0x7f5a48077b80 2024-04-16T07:11:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.346+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a48114b10 con 0x7f5a48077b80 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.347+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f5a3c019770 con 0x7f5a48077b80 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.347+0000 7f5a44c00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.347+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a30039460 con 0x7f5a300368a0 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.348+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5a3c049fe0 con 0x7f5a48077b80 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.348+0000 7f5a47e00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.348+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.549+0000 7f5a47e00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.549+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.950+0000 7f5a47e00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:14.950+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:15.751+0000 7f5a47e00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:15.751+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:17.353+0000 7f5a47e00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2299982413 2024-04-16T07:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:17.353+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:19.936+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 6) v1 ==== 43143+0+0 (secure 0 0 0) 0x7f5a3c0193c0 con 0x7f5a48077b80 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:19.936+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 msgr2=0x7f5a30038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:19.936+0000 7f5a44c00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.936+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 7) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f5a3c01a930 con 0x7f5a48077b80 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.937+0000 7f5a44c00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 0x7f5a3003c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.937+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a30039460 con 0x7f5a30039ff0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.940+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 0x7f5a3003c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.943+0000 7f5a47e00700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 0x7f5a3003c3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5a340039d0 tx=0x7f5a34009350 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.946+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f5a30039460 con 0x7f5a30039ff0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.953+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5a48114430 con 0x7f5a30039ff0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.953+0000 7f5a44c00700 1 -- 172.21.15.26:0/3820978370 <== mgr.14116 v2:172.21.15.26:6800/3603264388 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5a48114430 con 0x7f5a30039ff0 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 msgr2=0x7f5a3003c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 0x7f5a3003c3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5a340039d0 tx=0x7f5a34009350 comp rx=0 tx=0).stop 2024-04-16T07:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 msgr2=0x7f5a48113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c006fd0 tx=0x7f5a3c009250 comp rx=0 tx=0).stop 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 shutdown_connections 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f5a30039ff0 0x7f5a3003c3d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a48077b80 0x7f5a48113860 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 --2- 172.21.15.26:0/3820978370 >> [v2:172.21.15.26:6800/2299982413,v1:172.21.15.26:6801/2299982413] conn(0x7f5a300368a0 0x7f5a30038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.954+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 >> 172.21.15.26:0/3820978370 conn(0x7f5a480fe0a0 msgr2=0x7f5a480feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.955+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 shutdown_connections 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:20.955+0000 7f5a4f800700 1 -- 172.21.15.26:0/3820978370 wait complete. 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 5 is available 2024-04-16T07:11:21.746 INFO:teuthology.orchestra.run.smithi026.stdout:Setting orchestrator backend to cephadm... 2024-04-16T07:11:22.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:21 smithi026 ceph-mon[28805]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T07:11:22.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:21 smithi026 ceph-mon[28805]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T07:11:22.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:21 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:22.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:21 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: [16/Apr/2024:07:11:21] ENGINE Bus STARTING 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: [16/Apr/2024:07:11:21] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: [16/Apr/2024:07:11:21] ENGINE Bus STARTED 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: mgrmap e8: smithi026.tawexg(active, since 2s) 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:23.231 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:22 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:23.243 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.355+0000 7fb7c4a00700 1 Processor -- start 2024-04-16T07:11:23.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.355+0000 7fb7c4a00700 1 -- start start 2024-04-16T07:11:23.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.356+0000 7fb7c4a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:23.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.356+0000 7fb7c4a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0077f60 con 0x7fb7c0077b50 2024-04-16T07:11:23.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.356+0000 7fb7bde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:23.245 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.356+0000 7fb7bde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58354/0 (socket says 172.21.15.26:58354) 2024-04-16T07:11:23.245 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.356+0000 7fb7bde00700 1 -- 172.21.15.26:0/484719744 learned_addr learned my addr 172.21.15.26:0/484719744 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:23.245 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.357+0000 7fb7bde00700 1 -- 172.21.15.26:0/484719744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0076690 con 0x7fb7c0077b50 2024-04-16T07:11:23.246 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.357+0000 7fb7bde00700 1 --2- 172.21.15.26:0/484719744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00098e0 tx=0x7fb7b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0790ebc35b14d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:23.246 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.357+0000 7fb7bca00700 1 -- 172.21.15.26:0/484719744 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b000b860 con 0x7fb7c0077b50 2024-04-16T07:11:23.246 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7bca00700 1 -- 172.21.15.26:0/484719744 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb7b000be50 con 0x7fb7c0077b50 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7bca00700 1 -- 172.21.15.26:0/484719744 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b0010380 con 0x7fb7c0077b50 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 -- 172.21.15.26:0/484719744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 msgr2=0x7fb7c0076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/484719744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00098e0 tx=0x7fb7b0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 -- 172.21.15.26:0/484719744 shutdown_connections 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/484719744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0076150 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:23.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 -- 172.21.15.26:0/484719744 >> 172.21.15.26:0/484719744 conn(0x7fb7c00fe110 msgr2=0x7fb7c0100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 -- 172.21.15.26:0/484719744 shutdown_connections 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.358+0000 7fb7c4a00700 1 -- 172.21.15.26:0/484719744 wait complete. 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.359+0000 7fb7c4a00700 1 Processor -- start 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.359+0000 7fb7c4a00700 1 -- start start 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7c4a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7c4a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0113db0 con 0x7fb7c0077b50 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7bde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7bde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58356/0 (socket says 172.21.15.26:58356) 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7bde00700 1 -- 172.21.15.26:0/1778553215 learned_addr learned my addr 172.21.15.26:0/1778553215 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.360+0000 7fb7bde00700 1 -- 172.21.15.26:0/1778553215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b00094f0 con 0x7fb7c0077b50 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7bde00700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b001a7d0 con 0x7fb7c0077b50 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb7b001adc0 con 0x7fb7c0077b50 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c0113fb0 con 0x7fb7c0077b50 2024-04-16T07:11:23.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b0019e00 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.361+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c0114b20 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.362+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 7) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fb7b00193c0 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.362+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c005f5f0 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.363+0000 7fb7b6000700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 0x7fb7a4038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.363+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7b004a0a0 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.363+0000 7fb7bd400700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 0x7fb7a4038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.364+0000 7fb7bd400700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 0x7fb7a4038c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009990 tx=0x7fb7ac006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.369+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb7b007c250 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.579+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb7b0025400 con 0x7fb7c0077b50 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.601+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7c00ff2d0 con 0x7fb7a4036770 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.620+0000 7fb7b6000700 1 -- 172.21.15.26:0/1778553215 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb7c00ff2d0 con 0x7fb7a4036770 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 msgr2=0x7fb7a4038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 0x7fb7a4038c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009990 tx=0x7fb7ac006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 msgr2=0x7fb7c0113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:23.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 shutdown_connections 2024-04-16T07:11:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb7a4036770 0x7fb7a4038c20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 --2- 172.21.15.26:0/1778553215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7c0077b50 0x7fb7c0113870 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.623+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 >> 172.21.15.26:0/1778553215 conn(0x7fb7c00fe110 msgr2=0x7fb7c00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.624+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 shutdown_connections 2024-04-16T07:11:23.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:22.624+0000 7fb7c4a00700 1 -- 172.21.15.26:0/1778553215 wait complete. 2024-04-16T07:11:24.113 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:23 smithi026 ceph-mon[28805]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.859+0000 7f3a55e00700 1 Processor -- start 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.859+0000 7f3a55e00700 1 -- start start 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.860+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.860+0000 7f3a55e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a50103200 con 0x7f3a501028b0 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.860+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.860+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58358/0 (socket says 172.21.15.26:58358) 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.860+0000 7f3a4ea00700 1 -- 172.21.15.26:0/376219387 learned_addr learned my addr 172.21.15.26:0/376219387 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.861+0000 7f3a4ea00700 1 -- 172.21.15.26:0/376219387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50103340 con 0x7f3a501028b0 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.861+0000 7f3a4ea00700 1 --2- 172.21.15.26:0/376219387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098e0 tx=0x7f3a40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38db11c4896b6deb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.861+0000 7f3a4d600700 1 -- 172.21.15.26:0/376219387 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a4000b860 con 0x7f3a501028b0 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.861+0000 7f3a4d600700 1 -- 172.21.15.26:0/376219387 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f3a4000be50 con 0x7f3a501028b0 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.861+0000 7f3a4d600700 1 -- 172.21.15.26:0/376219387 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a40010380 con 0x7f3a501028b0 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.862+0000 7f3a55e00700 1 -- 172.21.15.26:0/376219387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 msgr2=0x7f3a50102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.862+0000 7f3a55e00700 1 --2- 172.21.15.26:0/376219387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098e0 tx=0x7f3a40009250 comp rx=0 tx=0).stop 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.862+0000 7f3a55e00700 1 -- 172.21.15.26:0/376219387 shutdown_connections 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.862+0000 7f3a55e00700 1 --2- 172.21.15.26:0/376219387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50102cc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.862+0000 7f3a55e00700 1 -- 172.21.15.26:0/376219387 >> 172.21.15.26:0/376219387 conn(0x7f3a500fe070 msgr2=0x7f3a50100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.863+0000 7f3a55e00700 1 -- 172.21.15.26:0/376219387 shutdown_connections 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.863+0000 7f3a55e00700 1 -- 172.21.15.26:0/376219387 wait complete. 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.863+0000 7f3a55e00700 1 Processor -- start 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a55e00700 1 -- start start 2024-04-16T07:11:24.827 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a55e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a50111ac0 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58362/0 (socket says 172.21.15.26:58362) 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.864+0000 7f3a4ea00700 1 -- 172.21.15.26:0/3476652016 learned_addr learned my addr 172.21.15.26:0/3476652016 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a4ea00700 1 -- 172.21.15.26:0/3476652016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400094f0 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a4ea00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a4001a870 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f3a4001ae60 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a50111cc0 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.865+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a400193c0 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.866+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a50112830 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.867+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5005f5f0 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.867+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f3a40022770 con 0x7f3a501028b0 2024-04-16T07:11:24.828 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.867+0000 7f3a46a00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 0x7f3a34038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:24.830 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.868+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3a4004a510 con 0x7f3a501028b0 2024-04-16T07:11:24.830 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.868+0000 7f3a4e000700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 0x7f3a34038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:24.830 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.869+0000 7f3a4e000700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 0x7f3a34038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009990 tx=0x7f3a3c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:23.873+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3a40010bd0 con 0x7f3a501028b0 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.092+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a500ff180 con 0x7f3a340368a0 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.094+0000 7f3a46a00700 1 -- 172.21.15.26:0/3476652016 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3a500ff180 con 0x7f3a340368a0 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 msgr2=0x7f3a34038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 0x7f3a34038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009990 tx=0x7f3a3c006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 msgr2=0x7f3a50111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a40009250 comp rx=0 tx=0).stop 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 shutdown_connections 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f3a340368a0 0x7f3a34038d50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 --2- 172.21.15.26:0/3476652016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a501028b0 0x7f3a50111580 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 >> 172.21.15.26:0/3476652016 conn(0x7f3a500fe070 msgr2=0x7f3a500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.096+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 shutdown_connections 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:24.097+0000 7f3a55e00700 1 -- 172.21.15.26:0/3476652016 wait complete. 2024-04-16T07:11:24.831 INFO:teuthology.orchestra.run.smithi026.stdout:Generating ssh key... 2024-04-16T07:11:25.122 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:24 smithi026 ceph-mon[28805]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:26.685 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.417+0000 7fdf87a00700 1 Processor -- start 2024-04-16T07:11:26.686 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.417+0000 7fdf87a00700 1 -- start start 2024-04-16T07:11:26.686 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.418+0000 7fdf87a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:26.686 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.418+0000 7fdf87a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80103250 con 0x7fdf80102900 2024-04-16T07:11:26.686 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.418+0000 7fdf85000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.418+0000 7fdf85000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58366/0 (socket says 172.21.15.26:58366) 2024-04-16T07:11:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.418+0000 7fdf85000700 1 -- 172.21.15.26:0/314488115 learned_addr learned my addr 172.21.15.26:0/314488115 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.419+0000 7fdf85000700 1 -- 172.21.15.26:0/314488115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80103390 con 0x7fdf80102900 2024-04-16T07:11:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.419+0000 7fdf85000700 1 --2- 172.21.15.26:0/314488115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098e0 tx=0x7fdf74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32044b6ca8046d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:26.688 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.419+0000 7fdf7f400700 1 -- 172.21.15.26:0/314488115 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7400b860 con 0x7fdf80102900 2024-04-16T07:11:26.688 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf7f400700 1 -- 172.21.15.26:0/314488115 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fdf7400be50 con 0x7fdf80102900 2024-04-16T07:11:26.688 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf7f400700 1 -- 172.21.15.26:0/314488115 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf74010380 con 0x7fdf80102900 2024-04-16T07:11:26.688 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf87a00700 1 -- 172.21.15.26:0/314488115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 msgr2=0x7fdf80102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:26.688 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf87a00700 1 --2- 172.21.15.26:0/314488115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098e0 tx=0x7fdf74009250 comp rx=0 tx=0).stop 2024-04-16T07:11:26.689 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf87a00700 1 -- 172.21.15.26:0/314488115 shutdown_connections 2024-04-16T07:11:26.689 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.420+0000 7fdf87a00700 1 --2- 172.21.15.26:0/314488115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80102d10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:26.689 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.421+0000 7fdf87a00700 1 -- 172.21.15.26:0/314488115 >> 172.21.15.26:0/314488115 conn(0x7fdf800fe0a0 msgr2=0x7fdf801004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:26.689 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.421+0000 7fdf87a00700 1 -- 172.21.15.26:0/314488115 shutdown_connections 2024-04-16T07:11:26.689 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.421+0000 7fdf87a00700 1 -- 172.21.15.26:0/314488115 wait complete. 2024-04-16T07:11:26.690 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.421+0000 7fdf87a00700 1 Processor -- start 2024-04-16T07:11:26.690 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf87a00700 1 -- start start 2024-04-16T07:11:26.690 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf87a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:26.690 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf87a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80113d60 con 0x7fdf80102900 2024-04-16T07:11:26.690 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf85000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:26.691 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf85000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58372/0 (socket says 172.21.15.26:58372) 2024-04-16T07:11:26.691 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.422+0000 7fdf85000700 1 -- 172.21.15.26:0/4042872377 learned_addr learned my addr 172.21.15.26:0/4042872377 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:26.692 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf85000700 1 -- 172.21.15.26:0/4042872377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740094f0 con 0x7fdf80102900 2024-04-16T07:11:26.692 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf85000700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf74000c00 tx=0x7fdf74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:26.692 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7401a8a0 con 0x7fdf80102900 2024-04-16T07:11:26.692 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fdf7401ae90 con 0x7fdf80102900 2024-04-16T07:11:26.692 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf80113f60 con 0x7fdf80102900 2024-04-16T07:11:26.693 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.423+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf74019e70 con 0x7fdf80102900 2024-04-16T07:11:26.693 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.424+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf80114ad0 con 0x7fdf80102900 2024-04-16T07:11:26.693 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.425+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8005f5f0 con 0x7fdf80102900 2024-04-16T07:11:26.693 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.425+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fdf74019730 con 0x7fdf80102900 2024-04-16T07:11:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.426+0000 7fdf7cc00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 0x7fdf68038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.426+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdf7404a0f0 con 0x7fdf80102900 2024-04-16T07:11:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.426+0000 7fdf7fe00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 0x7fdf68038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.427+0000 7fdf7fe00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 0x7fdf68038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009990 tx=0x7fdf6c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.431+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdf74021b10 con 0x7fdf80102900 2024-04-16T07:11:26.695 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:25.644+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf800ff240 con 0x7fdf680368a0 2024-04-16T07:11:26.695 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.005+0000 7fdf7cc00700 1 -- 172.21.15.26:0/4042872377 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdf800ff240 con 0x7fdf680368a0 2024-04-16T07:11:26.695 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.008+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 msgr2=0x7fdf68038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:26.696 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.008+0000 7fdf87a00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 0x7fdf68038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009990 tx=0x7fdf6c006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:26.696 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.008+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 msgr2=0x7fdf80113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:26.696 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.008+0000 7fdf87a00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf74000c00 tx=0x7fdf74009250 comp rx=0 tx=0).stop 2024-04-16T07:11:26.696 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 shutdown_connections 2024-04-16T07:11:26.696 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdf680368a0 0x7fdf68038d50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:26.697 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 --2- 172.21.15.26:0/4042872377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf80102900 0x7fdf80113820 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:26.697 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 >> 172.21.15.26:0/4042872377 conn(0x7fdf800fe0a0 msgr2=0x7fdf800feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:26.697 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 shutdown_connections 2024-04-16T07:11:26.697 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:26.009+0000 7fdf87a00700 1 -- 172.21.15.26:0/4042872377 wait complete. 2024-04-16T07:11:27.247 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:26 smithi026 ceph-mon[28805]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:27.247 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:26 smithi026 ceph-mon[28805]: Generating ssh key... 2024-04-16T07:11:27.247 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:26 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:27.247 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:26 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:28.252 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWPFNAM+fKiKz6j2R3V81/tafo91vupp0KnQJh4J9h+QjqY0Hofp4Cx9mSI6+ift7FAkj2K4wk+ab2IEmTaCE6HPKh0nX1QkJ4gc3Za/x3cpmsiFoNR66t1TWTF/ggJHomhF7vKK3B1eUENmVN3g6PC3GshqaCV8eDnrLEd3v2IxiLJxiVMmUV8mixy72TinCj6fMG2/LUxpzp3jtqSRvQxnAk6P1rsPE2vKn65yvjmd7POD2tkkNigs9U7r/meluij6pDR5wy9Iqol9jtHTq2NK9vPRLgRrQWq1pvmbq4bb3djbnPIMaMwnBTBvA0FE992anAqiJYANKFcX1RLhHpVxsADTm/xixGKcMkazyGOQr0MtLYy//RnF8AAQ/vvjyTjnGV1XCk9Ch7GdKjEebbUMH8iQk3nasbE2MgX7XFzus5SbHewWElS3WT0UxjLLt8ujMox00v9M2S/bS84s4BZL/nq08n3Pq470I5S4oAdr+dVMD28mR7+39xAf0XFdk= ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:11:28.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.312+0000 7f7be8400700 1 Processor -- start 2024-04-16T07:11:28.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be8400700 1 -- start start 2024-04-16T07:11:28.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be8400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:28.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be8400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be007a110 con 0x7f7be0079730 2024-04-16T07:11:28.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58388/0 (socket says 172.21.15.26:58388) 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.313+0000 7f7be5a00700 1 -- 172.21.15.26:0/2185546182 learned_addr learned my addr 172.21.15.26:0/2185546182 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.314+0000 7f7be5a00700 1 -- 172.21.15.26:0/2185546182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be007a250 con 0x7f7be0079730 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.314+0000 7f7be5a00700 1 --2- 172.21.15.26:0/2185546182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40098e0 tx=0x7f7bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fa80ae0c448e31 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.314+0000 7f7bdfe00700 1 -- 172.21.15.26:0/2185546182 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd400b860 con 0x7f7be0079730 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7bdfe00700 1 -- 172.21.15.26:0/2185546182 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7bd400be50 con 0x7f7be0079730 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7bdfe00700 1 -- 172.21.15.26:0/2185546182 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd4010380 con 0x7f7be0079730 2024-04-16T07:11:28.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7be8400700 1 -- 172.21.15.26:0/2185546182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 msgr2=0x7f7be0079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:28.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7be8400700 1 --2- 172.21.15.26:0/2185546182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40098e0 tx=0x7f7bd4009250 comp rx=0 tx=0).stop 2024-04-16T07:11:28.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7be8400700 1 -- 172.21.15.26:0/2185546182 shutdown_connections 2024-04-16T07:11:28.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7be8400700 1 --2- 172.21.15.26:0/2185546182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0079b40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.315+0000 7f7be8400700 1 -- 172.21.15.26:0/2185546182 >> 172.21.15.26:0/2185546182 conn(0x7f7be0078320 msgr2=0x7f7be0078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.316+0000 7f7be8400700 1 -- 172.21.15.26:0/2185546182 shutdown_connections 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.316+0000 7f7be8400700 1 -- 172.21.15.26:0/2185546182 wait complete. 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.316+0000 7f7be8400700 1 Processor -- start 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be8400700 1 -- start start 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be8400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be8400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be01180d0 con 0x7f7be0079730 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58390/0 (socket says 172.21.15.26:58390) 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.317+0000 7f7be5a00700 1 -- 172.21.15.26:0/3673434395 learned_addr learned my addr 172.21.15.26:0/3673434395 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7be5a00700 1 -- 172.21.15.26:0/3673434395 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd40094f0 con 0x7f7be0079730 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7be5a00700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400ba90 tx=0x7f7bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:28.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd401a8a0 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7bd401ae90 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd4019e80 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be01182d0 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.318+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be0118e40 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.319+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f7bd4018460 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.320+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be005f5f0 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.320+0000 7f7bdd600700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 0x7f7bc8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.320+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7bd404a0a0 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.320+0000 7f7be5000700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 0x7f7bc8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.321+0000 7f7be5000700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 0x7f7bc8038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009990 tx=0x7f7bcc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.325+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7bd4010560 con 0x7f7be0079730 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.544+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7be01199c0 con 0x7f7bc80368a0 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.546+0000 7f7bdd600700 1 -- 172.21.15.26:0/3673434395 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7be01199c0 con 0x7f7bc80368a0 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 msgr2=0x7f7bc8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:28.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 0x7f7bc8038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009990 tx=0x7f7bcc006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 msgr2=0x7f7be0117b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400ba90 tx=0x7f7bd4009250 comp rx=0 tx=0).stop 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 shutdown_connections 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7bc80368a0 0x7f7bc8038d50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 --2- 172.21.15.26:0/3673434395 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7be0079730 0x7f7be0117b90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 >> 172.21.15.26:0/3673434395 conn(0x7f7be0078320 msgr2=0x7f7be0102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 shutdown_connections 2024-04-16T07:11:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:27.549+0000 7f7be8400700 1 -- 172.21.15.26:0/3673434395 wait complete. 2024-04-16T07:11:28.263 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-16T07:11:28.263 INFO:teuthology.orchestra.run.smithi026.stdout:Adding key to root@localhost authorized_keys... 2024-04-16T07:11:28.263 INFO:teuthology.orchestra.run.smithi026.stdout:Adding host smithi026... 2024-04-16T07:11:28.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:28 smithi026 ceph-mon[28805]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:29.352 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:29 smithi026 ceph-mon[28805]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:31.710 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:31 smithi026 ceph-mon[28805]: Deploying cephadm binary to smithi026 2024-04-16T07:11:34.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Added host 'smithi026' with addr '172.21.15.26' 2024-04-16T07:11:34.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.856+0000 7fb8ce400700 1 Processor -- start 2024-04-16T07:11:34.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.856+0000 7fb8ce400700 1 -- start start 2024-04-16T07:11:34.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.857+0000 7fb8ce400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.857+0000 7fb8ce400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c8105600 con 0x7fb8c8104c20 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.857+0000 7fb8c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.857+0000 7fb8c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45418/0 (socket says 172.21.15.26:45418) 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.857+0000 7fb8c7400700 1 -- 172.21.15.26:0/2222563727 learned_addr learned my addr 172.21.15.26:0/2222563727 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.858+0000 7fb8c7400700 1 -- 172.21.15.26:0/2222563727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8105740 con 0x7fb8c8104c20 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.858+0000 7fb8c7400700 1 --2- 172.21.15.26:0/2222563727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80098e0 tx=0x7fb8b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c236a3fc19bd504 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.858+0000 7fb8c6a00700 1 -- 172.21.15.26:0/2222563727 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b800b860 con 0x7fb8c8104c20 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.858+0000 7fb8c6a00700 1 -- 172.21.15.26:0/2222563727 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb8b800be50 con 0x7fb8c8104c20 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.858+0000 7fb8c6a00700 1 -- 172.21.15.26:0/2222563727 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b8010380 con 0x7fb8c8104c20 2024-04-16T07:11:34.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 -- 172.21.15.26:0/2222563727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 msgr2=0x7fb8c8105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 --2- 172.21.15.26:0/2222563727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80098e0 tx=0x7fb8b8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 -- 172.21.15.26:0/2222563727 shutdown_connections 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 --2- 172.21.15.26:0/2222563727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c8105030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 -- 172.21.15.26:0/2222563727 >> 172.21.15.26:0/2222563727 conn(0x7fb8c8078350 msgr2=0x7fb8c8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.859+0000 7fb8ce400700 1 -- 172.21.15.26:0/2222563727 shutdown_connections 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.860+0000 7fb8ce400700 1 -- 172.21.15.26:0/2222563727 wait complete. 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.860+0000 7fb8ce400700 1 Processor -- start 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.861+0000 7fb8ce400700 1 -- start start 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.861+0000 7fb8ce400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.861+0000 7fb8ce400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c8113d30 con 0x7fb8c8104c20 2024-04-16T07:11:34.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.861+0000 7fb8c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.862+0000 7fb8c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45424/0 (socket says 172.21.15.26:45424) 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.862+0000 7fb8c7400700 1 -- 172.21.15.26:0/1561586398 learned_addr learned my addr 172.21.15.26:0/1561586398 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.862+0000 7fb8c7400700 1 -- 172.21.15.26:0/1561586398 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80094f0 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.862+0000 7fb8c7400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8006fd0 tx=0x7fb8b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.863+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b8010800 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.863+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb8b8010df0 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.863+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8113f30 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.863+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b801ad80 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.863+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c8114aa0 con 0x7fb8c8104c20 2024-04-16T07:11:34.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.864+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb8b801a660 con 0x7fb8c8104c20 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.864+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c805f5f0 con 0x7fb8c8104c20 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.864+0000 7fb8bf400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 0x7fb8ac038cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.864+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb8b8049ff0 con 0x7fb8c8104c20 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.865+0000 7fb8bfe00700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 0x7fb8ac038cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.866+0000 7fb8bfe00700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 0x7fb8ac038cb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009990 tx=0x7fb8b0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:28.870+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb8b807c250 con 0x7fb8c8104c20 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:29.091+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8c8102b30 con 0x7fb8ac036800 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.350+0000 7fb8bf400700 1 -- 172.21.15.26:0/1561586398 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb8c8102b30 con 0x7fb8ac036800 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 msgr2=0x7fb8ac038cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 0x7fb8ac038cb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009990 tx=0x7fb8b0006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:34.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 msgr2=0x7fb8c81137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8006fd0 tx=0x7fb8b8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 shutdown_connections 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb8ac036800 0x7fb8ac038cb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 --2- 172.21.15.26:0/1561586398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8c8104c20 0x7fb8c81137f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 >> 172.21.15.26:0/1561586398 conn(0x7fb8c8078350 msgr2=0x7fb8c8102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.355+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 shutdown_connections 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:33.356+0000 7fb8ce400700 1 -- 172.21.15.26:0/1561586398 wait complete. 2024-04-16T07:11:34.116 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying mon service with default placement... 2024-04-16T07:11:34.672 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:34 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:34.672 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:34 smithi026 ceph-mon[28805]: Added host smithi026 2024-04-16T07:11:34.673 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:34 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:11:36.013 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.846+0000 7f2ff5a00700 1 Processor -- start 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.846+0000 7f2ff5a00700 1 -- start start 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.846+0000 7f2ff5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.846+0000 7f2ff5a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff007d440 con 0x7f2ff007cfa0 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.847+0000 7f2feea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.847+0000 7f2feea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45438/0 (socket says 172.21.15.26:45438) 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.847+0000 7f2feea00700 1 -- 172.21.15.26:0/3400190040 learned_addr learned my addr 172.21.15.26:0/3400190040 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.848+0000 7f2feea00700 1 -- 172.21.15.26:0/3400190040 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff007d580 con 0x7f2ff007cfa0 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.848+0000 7f2feea00700 1 --2- 172.21.15.26:0/3400190040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098e0 tx=0x7f2fe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f43f2f06ef70565 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.848+0000 7f2fed600700 1 -- 172.21.15.26:0/3400190040 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe000b860 con 0x7f2ff007cfa0 2024-04-16T07:11:36.014 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.848+0000 7f2fed600700 1 -- 172.21.15.26:0/3400190040 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f2fe000be50 con 0x7f2ff007cfa0 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.848+0000 7f2fed600700 1 -- 172.21.15.26:0/3400190040 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe0010380 con 0x7f2ff007cfa0 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3400190040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 msgr2=0x7f2ff010bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3400190040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098e0 tx=0x7f2fe0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3400190040 shutdown_connections 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3400190040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff007cfa0 0x7f2ff010bd00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3400190040 >> 172.21.15.26:0/3400190040 conn(0x7f2ff0078560 msgr2=0x7f2ff007a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3400190040 shutdown_connections 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.849+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3400190040 wait complete. 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.850+0000 7f2ff5a00700 1 Processor -- start 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.850+0000 7f2ff5a00700 1 -- start start 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2ff5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2ff5a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0118630 con 0x7f2ff0117ce0 2024-04-16T07:11:36.015 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2feea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2feea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45444/0 (socket says 172.21.15.26:45444) 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2feea00700 1 -- 172.21.15.26:0/3503145096 learned_addr learned my addr 172.21.15.26:0/3503145096 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.851+0000 7f2feea00700 1 -- 172.21.15.26:0/3503145096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe00094f0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2feea00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098b0 tx=0x7f2fe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe001a7a0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f2fe001ad90 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff0118830 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe0024cf0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.852+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff0119210 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.853+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f2fe00245c0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.853+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff004bed0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.854+0000 7f2fe6a00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 0x7f2fd4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:36.016 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.854+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2fe004a2b0 con 0x7f2ff0117ce0 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.854+0000 7f2fee000700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 0x7f2fd4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.855+0000 7f2fee000700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 0x7f2fd4038d00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009990 tx=0x7f2fdc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:34.859+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2fe0010560 con 0x7f2ff0117ce0 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.096+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ff007d580 con 0x7f2fd4036850 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.104+0000 7f2fe6a00700 1 -- 172.21.15.26:0/3503145096 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2ff007d580 con 0x7f2fd4036850 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 msgr2=0x7f2fd4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 0x7f2fd4038d00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009990 tx=0x7f2fdc006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 msgr2=0x7f2ff01180f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098b0 tx=0x7f2fe0009250 comp rx=0 tx=0).stop 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 shutdown_connections 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f2fd4036850 0x7f2fd4038d00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 --2- 172.21.15.26:0/3503145096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ff0117ce0 0x7f2ff01180f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:36.017 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.107+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 >> 172.21.15.26:0/3503145096 conn(0x7f2ff0078560 msgr2=0x7f2ff007a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:36.018 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.108+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 shutdown_connections 2024-04-16T07:11:36.018 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:35.108+0000 7f2ff5a00700 1 -- 172.21.15.26:0/3503145096 wait complete. 2024-04-16T07:11:36.018 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying mgr service with default placement... 2024-04-16T07:11:36.420 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:36 smithi026 ceph-mon[28805]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:36.420 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:36 smithi026 ceph-mon[28805]: Saving service mon spec with placement count:5 2024-04-16T07:11:36.420 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:36 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:37.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:37 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:37.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:37 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:38.600 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-16T07:11:38.600 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.643+0000 7fc369000700 1 Processor -- start 2024-04-16T07:11:38.601 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.643+0000 7fc369000700 1 -- start start 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.644+0000 7fc369000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.644+0000 7fc369000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc364103250 con 0x7fc364102900 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.644+0000 7fc361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.644+0000 7fc361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45454/0 (socket says 172.21.15.26:45454) 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.644+0000 7fc361e00700 1 -- 172.21.15.26:0/1633026938 learned_addr learned my addr 172.21.15.26:0/1633026938 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.645+0000 7fc361e00700 1 -- 172.21.15.26:0/1633026938 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc364103390 con 0x7fc364102900 2024-04-16T07:11:38.602 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.645+0000 7fc361e00700 1 --2- 172.21.15.26:0/1633026938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc3540098e0 tx=0x7fc354009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1e94be72f57b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.645+0000 7fc360a00700 1 -- 172.21.15.26:0/1633026938 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35400b860 con 0x7fc364102900 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.645+0000 7fc360a00700 1 -- 172.21.15.26:0/1633026938 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc35400be90 con 0x7fc364102900 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.645+0000 7fc360a00700 1 -- 172.21.15.26:0/1633026938 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3540103c0 con 0x7fc364102900 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.646+0000 7fc369000700 1 -- 172.21.15.26:0/1633026938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 msgr2=0x7fc364102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.646+0000 7fc369000700 1 --2- 172.21.15.26:0/1633026938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc3540098e0 tx=0x7fc354009250 comp rx=0 tx=0).stop 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.646+0000 7fc369000700 1 -- 172.21.15.26:0/1633026938 shutdown_connections 2024-04-16T07:11:38.606 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.646+0000 7fc369000700 1 --2- 172.21.15.26:0/1633026938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364102d10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.646+0000 7fc369000700 1 -- 172.21.15.26:0/1633026938 >> 172.21.15.26:0/1633026938 conn(0x7fc3640fe0a0 msgr2=0x7fc3641004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.647+0000 7fc369000700 1 -- 172.21.15.26:0/1633026938 shutdown_connections 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.647+0000 7fc369000700 1 -- 172.21.15.26:0/1633026938 wait complete. 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.647+0000 7fc369000700 1 Processor -- start 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc369000700 1 -- start start 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc369000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc369000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640780a0 con 0x7fc364102900 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45460/0 (socket says 172.21.15.26:45460) 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.648+0000 7fc361e00700 1 -- 172.21.15.26:0/2818313027 learned_addr learned my addr 172.21.15.26:0/2818313027 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc361e00700 1 -- 172.21.15.26:0/2818313027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3540094f0 con 0x7fc364102900 2024-04-16T07:11:38.607 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc361e00700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc354006b00 tx=0x7fc354009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:38.610 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc354010840 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc354010e70 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35401ae70 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc364076110 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.649+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc364076600 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.650+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fc354024480 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.651+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36404bed0 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.651+0000 7fc35a000700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 0x7fc348038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.651+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc35404ab70 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.651+0000 7fc361400700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 0x7fc348038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.652+0000 7fc361400700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 0x7fc348038d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc350009990 tx=0x7fc350006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.657+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc35402e440 con 0x7fc364102900 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.885+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc364076d10 con 0x7fc3480368a0 2024-04-16T07:11:38.611 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.892+0000 7fc35a000700 1 -- 172.21.15.26:0/2818313027 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc364076d10 con 0x7fc3480368a0 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 msgr2=0x7fc348038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 0x7fc348038d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc350009990 tx=0x7fc350006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 msgr2=0x7fc364077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc354006b00 tx=0x7fc354009250 comp rx=0 tx=0).stop 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 shutdown_connections 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fc3480368a0 0x7fc348038d50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 --2- 172.21.15.26:0/2818313027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc364102900 0x7fc364077b60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 >> 172.21.15.26:0/2818313027 conn(0x7fc3640fe0a0 msgr2=0x7fc3640feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 shutdown_connections 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:37.895+0000 7fc369000700 1 -- 172.21.15.26:0/2818313027 wait complete. 2024-04-16T07:11:38.612 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying crash service with default placement... 2024-04-16T07:11:39.027 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:38 smithi026 ceph-mon[28805]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:39.027 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:38 smithi026 ceph-mon[28805]: Saving service mgr spec with placement count:2 2024-04-16T07:11:39.027 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:38 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.179+0000 7f71ecc00700 1 Processor -- start 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.179+0000 7f71ecc00700 1 -- start start 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.180+0000 7f71ecc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.180+0000 7f71ecc00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8079170 con 0x7f71e8078d60 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.180+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.180+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58426/0 (socket says 172.21.15.26:58426) 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.180+0000 7f71e5e00700 1 -- 172.21.15.26:0/451924602 learned_addr learned my addr 172.21.15.26:0/451924602 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.181+0000 7f71e5e00700 1 -- 172.21.15.26:0/451924602 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8077800 con 0x7f71e8078d60 2024-04-16T07:11:40.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.181+0000 7f71e5e00700 1 --2- 172.21.15.26:0/451924602 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71d80098e0 tx=0x7f71d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796d682e95f1adbd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.181+0000 7f71e4a00700 1 -- 172.21.15.26:0/451924602 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d800b860 con 0x7f71e8078d60 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.181+0000 7f71e4a00700 1 -- 172.21.15.26:0/451924602 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71d800be90 con 0x7f71e8078d60 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.181+0000 7f71e4a00700 1 -- 172.21.15.26:0/451924602 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d80103c0 con 0x7f71e8078d60 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 -- 172.21.15.26:0/451924602 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 msgr2=0x7f71e80772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 --2- 172.21.15.26:0/451924602 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71d80098e0 tx=0x7f71d8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 -- 172.21.15.26:0/451924602 shutdown_connections 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 --2- 172.21.15.26:0/451924602 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e80772c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 -- 172.21.15.26:0/451924602 >> 172.21.15.26:0/451924602 conn(0x7f71e80fe050 msgr2=0x7f71e8100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 -- 172.21.15.26:0/451924602 shutdown_connections 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.182+0000 7f71ecc00700 1 -- 172.21.15.26:0/451924602 wait complete. 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.183+0000 7f71ecc00700 1 Processor -- start 2024-04-16T07:11:40.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.183+0000 7f71ecc00700 1 -- start start 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.183+0000 7f71ecc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.183+0000 7f71ecc00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8113d10 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.184+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.184+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58442/0 (socket says 172.21.15.26:58442) 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.184+0000 7f71e5e00700 1 -- 172.21.15.26:0/4227193204 learned_addr learned my addr 172.21.15.26:0/4227193204 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.184+0000 7f71e5e00700 1 -- 172.21.15.26:0/4227193204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80094f0 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.184+0000 7f71e5e00700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006d40 tx=0x7f71d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.185+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d8010840 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.185+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71d8010e70 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.185+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d8024ea0 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.185+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e8113f10 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.185+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8114a80 con 0x7f71e8078d60 2024-04-16T07:11:40.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.186+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f71d8024780 con 0x7f71e8078d60 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.186+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e805f5f0 con 0x7f71e8078d60 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.187+0000 7f71de000700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 0x7f71cc038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.187+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f71d804a720 con 0x7f71e8078d60 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.187+0000 7f71e5400700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 0x7f71cc038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.188+0000 7f71e5400700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 0x7f71cc038d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f71d4009990 tx=0x7f71d4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.192+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f71d802c930 con 0x7f71e8078d60 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.435+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f71e80ff130 con 0x7f71cc036850 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.443+0000 7f71de000700 1 -- 172.21.15.26:0/4227193204 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f71e80ff130 con 0x7f71cc036850 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 msgr2=0x7f71cc038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 0x7f71cc038d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f71d4009990 tx=0x7f71d4006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 msgr2=0x7f71e81137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:40.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006d40 tx=0x7f71d8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 shutdown_connections 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f71cc036850 0x7f71cc038d00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 --2- 172.21.15.26:0/4227193204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71e8078d60 0x7f71e81137d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 >> 172.21.15.26:0/4227193204 conn(0x7f71e80fe050 msgr2=0x7f71e80fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 shutdown_connections 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:39.446+0000 7f71ecc00700 1 -- 172.21.15.26:0/4227193204 wait complete. 2024-04-16T07:11:40.499 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying prometheus service with default placement... 2024-04-16T07:11:40.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:40 smithi026 ceph-mon[28805]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:40.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:40 smithi026 ceph-mon[28805]: Saving service crash spec with placement * 2024-04-16T07:11:40.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:40 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:40.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:40 smithi026 ceph-mon[28805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:40.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:40 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:42.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-16T07:11:42.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.088+0000 7feeab400700 1 Processor -- start 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.088+0000 7feeab400700 1 -- start start 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.089+0000 7feeab400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.089+0000 7feeab400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea4077ff0 con 0x7feea4077be0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.089+0000 7feea8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.090+0000 7feea8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58458/0 (socket says 172.21.15.26:58458) 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.090+0000 7feea8a00700 1 -- 172.21.15.26:0/448281021 learned_addr learned my addr 172.21.15.26:0/448281021 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.090+0000 7feea8a00700 1 -- 172.21.15.26:0/448281021 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40766d0 con 0x7feea4077be0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.090+0000 7feea8a00700 1 --2- 172.21.15.26:0/448281021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee980098e0 tx=0x7fee98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff72654d6104eaa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feea3400700 1 -- 172.21.15.26:0/448281021 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee9800b860 con 0x7feea4077be0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feea3400700 1 -- 172.21.15.26:0/448281021 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee9800be90 con 0x7feea4077be0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feea3400700 1 -- 172.21.15.26:0/448281021 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee980103c0 con 0x7feea4077be0 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feeab400700 1 -- 172.21.15.26:0/448281021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 msgr2=0x7feea4076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feeab400700 1 --2- 172.21.15.26:0/448281021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee980098e0 tx=0x7fee98009250 comp rx=0 tx=0).stop 2024-04-16T07:11:42.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feeab400700 1 -- 172.21.15.26:0/448281021 shutdown_connections 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feeab400700 1 --2- 172.21.15.26:0/448281021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4076190 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.091+0000 7feeab400700 1 -- 172.21.15.26:0/448281021 >> 172.21.15.26:0/448281021 conn(0x7feea40fe0a0 msgr2=0x7feea41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.092+0000 7feeab400700 1 -- 172.21.15.26:0/448281021 shutdown_connections 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.092+0000 7feeab400700 1 -- 172.21.15.26:0/448281021 wait complete. 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feeab400700 1 Processor -- start 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feeab400700 1 -- start start 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feeab400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feeab400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea4113dd0 con 0x7feea4077be0 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feea8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feea8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58470/0 (socket says 172.21.15.26:58470) 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.093+0000 7feea8a00700 1 -- 172.21.15.26:0/1749778799 learned_addr learned my addr 172.21.15.26:0/1749778799 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feea8a00700 1 -- 172.21.15.26:0/1749778799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980094f0 con 0x7feea4077be0 2024-04-16T07:11:42.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feea8a00700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee980105b0 tx=0x7fee98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee98010840 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee98010e70 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee9801ae30 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.094+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea4113fd0 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.095+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea4114b40 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.096+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea405f5f0 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.096+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fee9801a430 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.097+0000 7feea0c00700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 0x7fee8c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.097+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fee9804a250 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.097+0000 7feea3e00700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 0x7fee8c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.098+0000 7feea3e00700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 0x7fee8c038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee90009990 tx=0x7fee90006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.102+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fee9802b4d0 con 0x7feea4077be0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.343+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7feea40ff240 con 0x7fee8c0368a0 2024-04-16T07:11:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.349+0000 7feea0c00700 1 -- 172.21.15.26:0/1749778799 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7feea40ff240 con 0x7fee8c0368a0 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 msgr2=0x7fee8c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 0x7fee8c038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee90009990 tx=0x7fee90006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 msgr2=0x7feea4113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee980105b0 tx=0x7fee98009250 comp rx=0 tx=0).stop 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 shutdown_connections 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fee8c0368a0 0x7fee8c038d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 --2- 172.21.15.26:0/1749778799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feea4077be0 0x7feea4113890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 >> 172.21.15.26:0/1749778799 conn(0x7feea40fe0a0 msgr2=0x7feea40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 shutdown_connections 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:41.351+0000 7feeab400700 1 -- 172.21.15.26:0/1749778799 wait complete. 2024-04-16T07:11:42.112 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying grafana service with default placement... 2024-04-16T07:11:42.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:42 smithi026 ceph-mon[28805]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:42.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:42 smithi026 ceph-mon[28805]: Saving service prometheus spec with placement count:1 2024-04-16T07:11:42.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:42 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:42.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:42 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:42.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:42 smithi026 ceph-mon[28805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.712+0000 7f1a2d400700 1 Processor -- start 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.712+0000 7f1a2d400700 1 -- start start 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.712+0000 7f1a2d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.712+0000 7f1a2d400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28103250 con 0x7f1a28102900 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.713+0000 7f1a26800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.713+0000 7f1a26800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58474/0 (socket says 172.21.15.26:58474) 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.713+0000 7f1a26800700 1 -- 172.21.15.26:0/292560689 learned_addr learned my addr 172.21.15.26:0/292560689 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.713+0000 7f1a26800700 1 -- 172.21.15.26:0/292560689 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28103390 con 0x7f1a28102900 2024-04-16T07:11:43.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.714+0000 7f1a26800700 1 --2- 172.21.15.26:0/292560689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a180098e0 tx=0x7f1a18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a5766d15706ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.714+0000 7f1a25400700 1 -- 172.21.15.26:0/292560689 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1800b860 con 0x7f1a28102900 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.714+0000 7f1a25400700 1 -- 172.21.15.26:0/292560689 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a1800be90 con 0x7f1a28102900 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.714+0000 7f1a25400700 1 -- 172.21.15.26:0/292560689 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a180103c0 con 0x7f1a28102900 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 -- 172.21.15.26:0/292560689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 msgr2=0x7f1a28102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 --2- 172.21.15.26:0/292560689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a180098e0 tx=0x7f1a18009250 comp rx=0 tx=0).stop 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 -- 172.21.15.26:0/292560689 shutdown_connections 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 --2- 172.21.15.26:0/292560689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28102d10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 -- 172.21.15.26:0/292560689 >> 172.21.15.26:0/292560689 conn(0x7f1a280fe0a0 msgr2=0x7f1a281004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 -- 172.21.15.26:0/292560689 shutdown_connections 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.715+0000 7f1a2d400700 1 -- 172.21.15.26:0/292560689 wait complete. 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.716+0000 7f1a2d400700 1 Processor -- start 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.716+0000 7f1a2d400700 1 -- start start 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a2d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a2d400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28113dd0 con 0x7f1a28102900 2024-04-16T07:11:43.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a26800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a26800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58484/0 (socket says 172.21.15.26:58484) 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a26800700 1 -- 172.21.15.26:0/300248245 learned_addr learned my addr 172.21.15.26:0/300248245 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.717+0000 7f1a26800700 1 -- 172.21.15.26:0/300248245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180094f0 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a26800700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a18006b00 tx=0x7f1a18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a18010840 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a18010e70 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1801ae70 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a28113fd0 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.718+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a28114b40 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.720+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2805f5f0 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.720+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f1a1801a780 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.720+0000 7f1a1ea00700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 0x7f1a0c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.720+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1a1804ab90 con 0x7f1a28102900 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.720+0000 7f1a25e00700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 0x7f1a0c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:43.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.721+0000 7f1a25e00700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 0x7f1a0c038d00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009990 tx=0x7f1a14006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.726+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1a1801a430 con 0x7f1a28102900 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.976+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a280ff180 con 0x7f1a0c036850 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.986+0000 7f1a1ea00700 1 -- 172.21.15.26:0/300248245 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1a280ff180 con 0x7f1a0c036850 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.988+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 msgr2=0x7f1a0c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.988+0000 7f1a2d400700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 0x7f1a0c038d00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009990 tx=0x7f1a14006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.988+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 msgr2=0x7f1a28113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.988+0000 7f1a2d400700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a18006b00 tx=0x7f1a18009250 comp rx=0 tx=0).stop 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 shutdown_connections 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f1a0c036850 0x7f1a0c038d00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 --2- 172.21.15.26:0/300248245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a28102900 0x7f1a28113890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 >> 172.21.15.26:0/300248245 conn(0x7f1a280fe0a0 msgr2=0x7f1a280fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 shutdown_connections 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:42.989+0000 7f1a2d400700 1 -- 172.21.15.26:0/300248245 wait complete. 2024-04-16T07:11:43.651 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying node-exporter service with default placement... 2024-04-16T07:11:44.257 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:43 smithi026 ceph-mon[28805]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:44.257 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:43 smithi026 ceph-mon[28805]: Saving service grafana spec with placement count:1 2024-04-16T07:11:44.257 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:43 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:45.033 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:44 smithi026 ceph-mon[28805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:45.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-16T07:11:45.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.779+0000 7ff88c400700 1 Processor -- start 2024-04-16T07:11:45.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.779+0000 7ff88c400700 1 -- start start 2024-04-16T07:11:45.743 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.779+0000 7ff88c400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.779+0000 7ff88c400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff884105600 con 0x7ff884104c20 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.780+0000 7ff889a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.780+0000 7ff889a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58498/0 (socket says 172.21.15.26:58498) 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.780+0000 7ff889a00700 1 -- 172.21.15.26:0/339622907 learned_addr learned my addr 172.21.15.26:0/339622907 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.781+0000 7ff889a00700 1 -- 172.21.15.26:0/339622907 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884105740 con 0x7ff884104c20 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.781+0000 7ff889a00700 1 --2- 172.21.15.26:0/339622907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8780098e0 tx=0x7ff878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107ed4f32ba239d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.781+0000 7ff883e00700 1 -- 172.21.15.26:0/339622907 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff87800b860 con 0x7ff884104c20 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.781+0000 7ff883e00700 1 -- 172.21.15.26:0/339622907 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff87800be90 con 0x7ff884104c20 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.781+0000 7ff883e00700 1 -- 172.21.15.26:0/339622907 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8780103c0 con 0x7ff884104c20 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 -- 172.21.15.26:0/339622907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 msgr2=0x7ff884105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 --2- 172.21.15.26:0/339622907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8780098e0 tx=0x7ff878009250 comp rx=0 tx=0).stop 2024-04-16T07:11:45.744 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 -- 172.21.15.26:0/339622907 shutdown_connections 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 --2- 172.21.15.26:0/339622907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884105030 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 -- 172.21.15.26:0/339622907 >> 172.21.15.26:0/339622907 conn(0x7ff884078350 msgr2=0x7ff884078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 -- 172.21.15.26:0/339622907 shutdown_connections 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.782+0000 7ff88c400700 1 -- 172.21.15.26:0/339622907 wait complete. 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.783+0000 7ff88c400700 1 Processor -- start 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.783+0000 7ff88c400700 1 -- start start 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.783+0000 7ff88c400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.783+0000 7ff88c400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff884113d40 con 0x7ff884104c20 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.784+0000 7ff889a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.784+0000 7ff889a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58510/0 (socket says 172.21.15.26:58510) 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.784+0000 7ff889a00700 1 -- 172.21.15.26:0/2771375312 learned_addr learned my addr 172.21.15.26:0/2771375312 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:45.745 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.784+0000 7ff889a00700 1 -- 172.21.15.26:0/2771375312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780094f0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.784+0000 7ff889a00700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff878011c10 tx=0x7ff878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.785+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff878010890 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.785+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff878010ec0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.785+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff884113f40 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.785+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8780243f0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.785+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff884114ab0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.786+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7ff8780109f0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.786+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88405f5f0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.786+0000 7ff881600700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 0x7ff86c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.786+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff87804a2a0 con 0x7ff884104c20 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.787+0000 7ff889000700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 0x7ff86c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:45.746 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.787+0000 7ff889000700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 0x7ff86c038d50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff870009990 tx=0x7ff870006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:44.792+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff87802b4d0 con 0x7ff884104c20 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.025+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff884102ad0 con 0x7ff86c0368a0 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.034+0000 7ff881600700 1 -- 172.21.15.26:0/2771375312 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff884102ad0 con 0x7ff86c0368a0 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 msgr2=0x7ff86c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 0x7ff86c038d50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff870009990 tx=0x7ff870006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 msgr2=0x7ff884113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff878011c10 tx=0x7ff878009250 comp rx=0 tx=0).stop 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 shutdown_connections 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7ff86c0368a0 0x7ff86c038d50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 --2- 172.21.15.26:0/2771375312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff884104c20 0x7ff884113800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.037+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 >> 172.21.15.26:0/2771375312 conn(0x7ff884078350 msgr2=0x7ff8841023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:45.747 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.038+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 shutdown_connections 2024-04-16T07:11:45.748 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:45.038+0000 7ff88c400700 1 -- 172.21.15.26:0/2771375312 wait complete. 2024-04-16T07:11:45.748 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying alertmanager service with default placement... 2024-04-16T07:11:46.317 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:46 smithi026 ceph-mon[28805]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:46.317 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:46 smithi026 ceph-mon[28805]: Saving service node-exporter spec with placement * 2024-04-16T07:11:46.317 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:46 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:47.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:47 smithi026 ceph-mon[28805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:47.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:47 smithi026 ceph-mon[28805]: from='mgr.14116 172.21.15.26:0/2164878249' entity='mgr.smithi026.tawexg' 2024-04-16T07:11:47.400 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.392+0000 7fb9e9800700 1 Processor -- start 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.392+0000 7fb9e9800700 1 -- start start 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.392+0000 7fb9e9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.392+0000 7fb9e9800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e407d420 con 0x7fb9e407ca40 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.393+0000 7fb9e2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.393+0000 7fb9e2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58524/0 (socket says 172.21.15.26:58524) 2024-04-16T07:11:47.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.393+0000 7fb9e2a00700 1 -- 172.21.15.26:0/2971638203 learned_addr learned my addr 172.21.15.26:0/2971638203 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.393+0000 7fb9e2a00700 1 -- 172.21.15.26:0/2971638203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e407d560 con 0x7fb9e407ca40 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e2a00700 1 --2- 172.21.15.26:0/2971638203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb9d40098e0 tx=0x7fb9d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fe3d9e73079f42 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e1600700 1 -- 172.21.15.26:0/2971638203 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d400b860 con 0x7fb9e407ca40 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e1600700 1 -- 172.21.15.26:0/2971638203 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9d400be90 con 0x7fb9e407ca40 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e1600700 1 -- 172.21.15.26:0/2971638203 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d40103c0 con 0x7fb9e407ca40 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e9800700 1 -- 172.21.15.26:0/2971638203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 msgr2=0x7fb9e407ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.394+0000 7fb9e9800700 1 --2- 172.21.15.26:0/2971638203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb9d40098e0 tx=0x7fb9d4009250 comp rx=0 tx=0).stop 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.395+0000 7fb9e9800700 1 -- 172.21.15.26:0/2971638203 shutdown_connections 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.395+0000 7fb9e9800700 1 --2- 172.21.15.26:0/2971638203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e407ce50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.395+0000 7fb9e9800700 1 -- 172.21.15.26:0/2971638203 >> 172.21.15.26:0/2971638203 conn(0x7fb9e4078350 msgr2=0x7fb9e407a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:47.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.395+0000 7fb9e9800700 1 -- 172.21.15.26:0/2971638203 shutdown_connections 2024-04-16T07:11:47.406 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.395+0000 7fb9e9800700 1 -- 172.21.15.26:0/2971638203 wait complete. 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e9800700 1 Processor -- start 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e9800700 1 -- start start 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e9800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4118150 con 0x7fb9e407ca40 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58538/0 (socket says 172.21.15.26:58538) 2024-04-16T07:11:47.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.396+0000 7fb9e2a00700 1 -- 172.21.15.26:0/3345926829 learned_addr learned my addr 172.21.15.26:0/3345926829 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9e2a00700 1 -- 172.21.15.26:0/3345926829 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40094f0 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9e2a00700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006b00 tx=0x7fb9d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d4010840 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9d4010e70 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e4118350 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.397+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d401ae70 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.398+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e4118ec0 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.399+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb9d401a780 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.399+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e405f5f0 con 0x7fb9e407ca40 2024-04-16T07:11:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.399+0000 7fb9daa00700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 0x7fb9c8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.399+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb9d404ab60 con 0x7fb9e407ca40 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.399+0000 7fb9e2000700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 0x7fb9c8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.400+0000 7fb9e2000700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 0x7fb9c8038d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009990 tx=0x7fb9d0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.404+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb9d401a430 con 0x7fb9e407ca40 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.630+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9e407a4d0 con 0x7fb9c80368a0 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.642+0000 7fb9daa00700 1 -- 172.21.15.26:0/3345926829 <== mgr.14116 v2:172.21.15.26:6800/3603264388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb9e407a4d0 con 0x7fb9c80368a0 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.644+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 msgr2=0x7fb9c8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.644+0000 7fb9e9800700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 0x7fb9c8038d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009990 tx=0x7fb9d0006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.644+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 msgr2=0x7fb9e4117c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.644+0000 7fb9e9800700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006b00 tx=0x7fb9d4009250 comp rx=0 tx=0).stop 2024-04-16T07:11:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 shutdown_connections 2024-04-16T07:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fb9c80368a0 0x7fb9c8038d50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 --2- 172.21.15.26:0/3345926829 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9e407ca40 0x7fb9e4117c10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 >> 172.21.15.26:0/3345926829 conn(0x7fb9e4078350 msgr2=0x7fb9e4079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 shutdown_connections 2024-04-16T07:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:46.645+0000 7fb9e9800700 1 -- 172.21.15.26:0/3345926829 wait complete. 2024-04-16T07:11:48.280 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:48 smithi026 ceph-mon[28805]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:11:48.281 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:48 smithi026 ceph-mon[28805]: Saving service alertmanager spec with placement count:1 2024-04-16T07:11:48.936 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.020+0000 7fd129400700 1 Processor -- start 2024-04-16T07:11:48.936 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd129400700 1 -- start start 2024-04-16T07:11:48.936 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd129400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:48.936 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd129400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd124103200 con 0x7fd1241028b0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd122800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd122800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53654/0 (socket says 172.21.15.26:53654) 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.021+0000 7fd122800700 1 -- 172.21.15.26:0/3275185401 learned_addr learned my addr 172.21.15.26:0/3275185401 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.022+0000 7fd122800700 1 -- 172.21.15.26:0/3275185401 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124103340 con 0x7fd1241028b0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.022+0000 7fd122800700 1 --2- 172.21.15.26:0/3275185401 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098e0 tx=0x7fd114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b714b28a8367520 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.022+0000 7fd121400700 1 -- 172.21.15.26:0/3275185401 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11400b860 con 0x7fd1241028b0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd121400700 1 -- 172.21.15.26:0/3275185401 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd11400be90 con 0x7fd1241028b0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd121400700 1 -- 172.21.15.26:0/3275185401 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1140103c0 con 0x7fd1241028b0 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 -- 172.21.15.26:0/3275185401 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 msgr2=0x7fd124102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 --2- 172.21.15.26:0/3275185401 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098e0 tx=0x7fd114009250 comp rx=0 tx=0).stop 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 -- 172.21.15.26:0/3275185401 shutdown_connections 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 --2- 172.21.15.26:0/3275185401 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd124102cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 -- 172.21.15.26:0/3275185401 >> 172.21.15.26:0/3275185401 conn(0x7fd1240fe070 msgr2=0x7fd124100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.023+0000 7fd129400700 1 -- 172.21.15.26:0/3275185401 shutdown_connections 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.024+0000 7fd129400700 1 -- 172.21.15.26:0/3275185401 wait complete. 2024-04-16T07:11:48.937 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.024+0000 7fd129400700 1 Processor -- start 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd129400700 1 -- start start 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd129400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd129400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd124113d30 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd122800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd122800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53670/0 (socket says 172.21.15.26:53670) 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.025+0000 7fd122800700 1 -- 172.21.15.26:0/1893730219 learned_addr learned my addr 172.21.15.26:0/1893730219 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd122800700 1 -- 172.21.15.26:0/1893730219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1140094f0 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd122800700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd1140090b0 tx=0x7fd114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd114010840 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd114010e70 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd114024e40 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd124113f30 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.026+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd124114aa0 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.027+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fd114024430 con 0x7fd1241028b0 2024-04-16T07:11:48.938 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.027+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12405f5f0 con 0x7fd1241028b0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.028+0000 7fd11aa00700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 0x7fd108038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.028+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd11404a810 con 0x7fd1241028b0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.028+0000 7fd121e00700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 0x7fd108038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.029+0000 7fd121e00700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 0x7fd108038d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd110009990 tx=0x7fd110006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.033+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd11402c910 con 0x7fd1241028b0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.252+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd12404bed0 con 0x7fd1241028b0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.260+0000 7fd11aa00700 1 -- 172.21.15.26:0/1893730219 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd11402c340 con 0x7fd1241028b0 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 msgr2=0x7fd108038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 0x7fd108038d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd110009990 tx=0x7fd110006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 msgr2=0x7fd1241137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd1140090b0 tx=0x7fd114009250 comp rx=0 tx=0).stop 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 shutdown_connections 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fd1080368a0 0x7fd108038d50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 --2- 172.21.15.26:0/1893730219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1241028b0 0x7fd1241137f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.263+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 >> 172.21.15.26:0/1893730219 conn(0x7fd1240fe070 msgr2=0x7fd1240fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:48.939 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.264+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 shutdown_connections 2024-04-16T07:11:48.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:48.264+0000 7fd129400700 1 -- 172.21.15.26:0/1893730219 wait complete. 2024-04-16T07:11:49.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:49 smithi026 ceph-mon[28805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:49.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:49 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1893730219' entity='client.admin' 2024-04-16T07:11:50.395 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.527+0000 7fdd7d000700 1 Processor -- start 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.527+0000 7fdd7d000700 1 -- start start 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.527+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.527+0000 7fdd7d000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7810ace0 con 0x7fdd7810a300 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.528+0000 7fdd77400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.528+0000 7fdd77400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53676/0 (socket says 172.21.15.26:53676) 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.528+0000 7fdd77400700 1 -- 172.21.15.26:0/4262307297 learned_addr learned my addr 172.21.15.26:0/4262307297 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.528+0000 7fdd77400700 1 -- 172.21.15.26:0/4262307297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7807c4e0 con 0x7fdd7810a300 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.528+0000 7fdd77400700 1 --2- 172.21.15.26:0/4262307297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd680098e0 tx=0x7fdd68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9d51db246071d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.529+0000 7fdd76a00700 1 -- 172.21.15.26:0/4262307297 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd6800b860 con 0x7fdd7810a300 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.529+0000 7fdd76a00700 1 -- 172.21.15.26:0/4262307297 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd6800be90 con 0x7fdd7810a300 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.529+0000 7fdd76a00700 1 -- 172.21.15.26:0/4262307297 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd680103c0 con 0x7fdd7810a300 2024-04-16T07:11:50.396 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.529+0000 7fdd7d000700 1 -- 172.21.15.26:0/4262307297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 msgr2=0x7fdd7810a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.529+0000 7fdd7d000700 1 --2- 172.21.15.26:0/4262307297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd680098e0 tx=0x7fdd68009250 comp rx=0 tx=0).stop 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.530+0000 7fdd7d000700 1 -- 172.21.15.26:0/4262307297 shutdown_connections 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.530+0000 7fdd7d000700 1 --2- 172.21.15.26:0/4262307297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd7810a710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.530+0000 7fdd7d000700 1 -- 172.21.15.26:0/4262307297 >> 172.21.15.26:0/4262307297 conn(0x7fdd78078010 msgr2=0x7fdd7807a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.530+0000 7fdd7d000700 1 -- 172.21.15.26:0/4262307297 shutdown_connections 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.530+0000 7fdd7d000700 1 -- 172.21.15.26:0/4262307297 wait complete. 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.531+0000 7fdd7d000700 1 Processor -- start 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.531+0000 7fdd7d000700 1 -- start start 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd7d000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78117ef0 con 0x7fdd7810a300 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd77400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd77400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53686/0 (socket says 172.21.15.26:53686) 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd77400700 1 -- 172.21.15.26:0/808876168 learned_addr learned my addr 172.21.15.26:0/808876168 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:50.397 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.532+0000 7fdd77400700 1 -- 172.21.15.26:0/808876168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680094f0 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd77400700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd680090b0 tx=0x7fdd68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd68010840 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd68010e70 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd68024e40 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd781180f0 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.533+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd78118c60 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.534+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd78118580 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.534+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fdd68024430 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.535+0000 7fdd6f400700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 0x7fdd5c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.535+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdd6804a810 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.535+0000 7fdd6fe00700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 0x7fdd5c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.536+0000 7fdd6fe00700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 0x7fdd5c038d00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009990 tx=0x7fdd60006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.541+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdd6802c910 con 0x7fdd7810a300 2024-04-16T07:11:50.398 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.758+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fdd7805f5f0 con 0x7fdd7810a300 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.766+0000 7fdd6f400700 1 -- 172.21.15.26:0/808876168 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fdd6802c340 con 0x7fdd7810a300 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.768+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 msgr2=0x7fdd5c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.768+0000 7fdd7d000700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 0x7fdd5c038d00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009990 tx=0x7fdd60006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.768+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 msgr2=0x7fdd781179b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.768+0000 7fdd7d000700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd680090b0 tx=0x7fdd68009250 comp rx=0 tx=0).stop 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 shutdown_connections 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7fdd5c036850 0x7fdd5c038d00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 --2- 172.21.15.26:0/808876168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd7810a300 0x7fdd781179b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 >> 172.21.15.26:0/808876168 conn(0x7fdd78078010 msgr2=0x7fdd780789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 shutdown_connections 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:49.769+0000 7fdd7d000700 1 -- 172.21.15.26:0/808876168 wait complete. 2024-04-16T07:11:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling the dashboard module... 2024-04-16T07:11:51.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:50 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/808876168' entity='client.admin' 2024-04-16T07:11:51.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:50 smithi026 ceph-mon[28805]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:11:52.159 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:51 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2198914434' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-16T07:11:52.498 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.042+0000 7f0685000700 1 Processor -- start 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.042+0000 7f0685000700 1 -- start start 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.043+0000 7f0685000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.043+0000 7f0685000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068007d420 con 0x7f068007ca40 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.043+0000 7f067de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.043+0000 7f067de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53688/0 (socket says 172.21.15.26:53688) 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.043+0000 7f067de00700 1 -- 172.21.15.26:0/1631698187 learned_addr learned my addr 172.21.15.26:0/1631698187 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.044+0000 7f067de00700 1 -- 172.21.15.26:0/1631698187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068007d560 con 0x7f068007ca40 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.044+0000 7f067de00700 1 --2- 172.21.15.26:0/1631698187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06700098e0 tx=0x7f0670009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6c0e3aa0861901 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.044+0000 7f067ca00700 1 -- 172.21.15.26:0/1631698187 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f067000b860 con 0x7f068007ca40 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f067ca00700 1 -- 172.21.15.26:0/1631698187 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f067000be90 con 0x7f068007ca40 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f067ca00700 1 -- 172.21.15.26:0/1631698187 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06700103c0 con 0x7f068007ca40 2024-04-16T07:11:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f0685000700 1 -- 172.21.15.26:0/1631698187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 msgr2=0x7f068007ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f0685000700 1 --2- 172.21.15.26:0/1631698187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06700098e0 tx=0x7f0670009250 comp rx=0 tx=0).stop 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f0685000700 1 -- 172.21.15.26:0/1631698187 shutdown_connections 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.045+0000 7f0685000700 1 --2- 172.21.15.26:0/1631698187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f068007ce50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.046+0000 7f0685000700 1 -- 172.21.15.26:0/1631698187 >> 172.21.15.26:0/1631698187 conn(0x7f0680078350 msgr2=0x7f068007a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.046+0000 7f0685000700 1 -- 172.21.15.26:0/1631698187 shutdown_connections 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.046+0000 7f0685000700 1 -- 172.21.15.26:0/1631698187 wait complete. 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.047+0000 7f0685000700 1 Processor -- start 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.047+0000 7f0685000700 1 -- start start 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f0685000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f0685000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0680118150 con 0x7f068007ca40 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f067de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:52.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f067de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53690/0 (socket says 172.21.15.26:53690) 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f067de00700 1 -- 172.21.15.26:0/2198914434 learned_addr learned my addr 172.21.15.26:0/2198914434 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.048+0000 7f067de00700 1 -- 172.21.15.26:0/2198914434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06700094f0 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f067de00700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0670006b00 tx=0x7f0670009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0670010840 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0670010e70 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0680118350 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f067001ae70 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.049+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0680118ec0 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.050+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f06700109a0 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.050+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068005f5f0 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.051+0000 7f0676000700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 0x7f0664038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.051+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f067004ab10 con 0x7f068007ca40 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.051+0000 7f067d400700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 0x7f0664038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:52.507 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.052+0000 7f067d400700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 0x7f0664038d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f066c009990 tx=0x7f066c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.056+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f067001a780 con 0x7f068007ca40 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.330+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f068004bed0 con 0x7f068007ca40 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.776+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f067001a430 con 0x7f068007ca40 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.779+0000 7f0676000700 1 -- 172.21.15.26:0/2198914434 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0670017ed0 con 0x7f068007ca40 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 msgr2=0x7f0664038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 0x7f0664038d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f066c009990 tx=0x7f066c006e40 comp rx=0 tx=0).stop 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 msgr2=0x7f0680117c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0670006b00 tx=0x7f0670009250 comp rx=0 tx=0).stop 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 shutdown_connections 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f0664036850 0x7f0664038d00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 --2- 172.21.15.26:0/2198914434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f068007ca40 0x7f0680117c10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.782+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 >> 172.21.15.26:0/2198914434 conn(0x7f0680078350 msgr2=0x7f0680079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.783+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 shutdown_connections 2024-04-16T07:11:52.508 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:51.783+0000 7f0685000700 1 -- 172.21.15.26:0/2198914434 wait complete. 2024-04-16T07:11:53.075 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:52 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2198914434' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-16T07:11:53.075 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:52 smithi026 ceph-mon[28805]: mgrmap e9: smithi026.tawexg(active, since 31s) 2024-04-16T07:11:53.927 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:11:53 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2175533162' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "smithi026.tawexg", 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.123+0000 7f4eee400700 1 Processor -- start 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.123+0000 7f4eee400700 1 -- start start 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.124+0000 7f4eee400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:54.281 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.124+0000 7f4eee400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8103250 con 0x7f4ee8102900 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.124+0000 7f4ee7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.124+0000 7f4ee7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53716/0 (socket says 172.21.15.26:53716) 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.124+0000 7f4ee7400700 1 -- 172.21.15.26:0/2841607984 learned_addr learned my addr 172.21.15.26:0/2841607984 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.125+0000 7f4ee7400700 1 -- 172.21.15.26:0/2841607984 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8103390 con 0x7f4ee8102900 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.125+0000 7f4ee7400700 1 --2- 172.21.15.26:0/2841607984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098e0 tx=0x7f4ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261d42a67d912206 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.125+0000 7f4ee6000700 1 -- 172.21.15.26:0/2841607984 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed800b860 con 0x7f4ee8102900 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.126+0000 7f4ee6000700 1 -- 172.21.15.26:0/2841607984 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed800be90 con 0x7f4ee8102900 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.126+0000 7f4ee6000700 1 -- 172.21.15.26:0/2841607984 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed80103c0 con 0x7f4ee8102900 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.126+0000 7f4eee400700 1 -- 172.21.15.26:0/2841607984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 msgr2=0x7f4ee8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.126+0000 7f4eee400700 1 --2- 172.21.15.26:0/2841607984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098e0 tx=0x7f4ed8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:54.282 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.127+0000 7f4eee400700 1 -- 172.21.15.26:0/2841607984 shutdown_connections 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.127+0000 7f4eee400700 1 --2- 172.21.15.26:0/2841607984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8102d10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.127+0000 7f4eee400700 1 -- 172.21.15.26:0/2841607984 >> 172.21.15.26:0/2841607984 conn(0x7f4ee80fe0a0 msgr2=0x7f4ee81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.127+0000 7f4eee400700 1 -- 172.21.15.26:0/2841607984 shutdown_connections 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.127+0000 7f4eee400700 1 -- 172.21.15.26:0/2841607984 wait complete. 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.128+0000 7f4eee400700 1 Processor -- start 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.128+0000 7f4eee400700 1 -- start start 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.128+0000 7f4eee400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.128+0000 7f4eee400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8113d90 con 0x7f4ee8102900 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.129+0000 7f4ee7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.129+0000 7f4ee7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53724/0 (socket says 172.21.15.26:53724) 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.129+0000 7f4ee7400700 1 -- 172.21.15.26:0/2175533162 learned_addr learned my addr 172.21.15.26:0/2175533162 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:11:54.283 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.129+0000 7f4ee7400700 1 -- 172.21.15.26:0/2175533162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80094f0 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.129+0000 7f4ee7400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8006b00 tx=0x7f4ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.130+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed8010840 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.130+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed8010e70 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.130+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee8113f90 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.130+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed801ae70 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.130+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee8114b00 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.131+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f4ed80244d0 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.132+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee805f5f0 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.132+0000 7f4edf400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.132+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4ed804adc0 con 0x7f4ee8102900 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.132+0000 7f4ee6a00700 1 -- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.132+0000 7f4ee6a00700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:11:54.284 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.138+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4ed8018350 con 0x7f4ee8102900 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.333+0000 7f4ee6a00700 1 -- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.333+0000 7f4ee6a00700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.409+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4ee8114ea0 con 0x7f4ee8102900 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.409+0000 7f4edf400700 1 -- 172.21.15.26:0/2175533162 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f4ed802cb80 con 0x7f4ee8102900 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 msgr2=0x7f4ee8113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8006b00 tx=0x7f4ed8009250 comp rx=0 tx=0).stop 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 shutdown_connections 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 --2- 172.21.15.26:0/2175533162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ee8102900 0x7f4ee8113850 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 >> 172.21.15.26:0/2175533162 conn(0x7f4ee80fe0a0 msgr2=0x7f4ee80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 shutdown_connections 2024-04-16T07:11:54.285 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:53.412+0000 7f4eee400700 1 -- 172.21.15.26:0/2175533162 wait complete. 2024-04-16T07:11:54.286 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-04-16T07:11:54.286 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 9... 2024-04-16T07:12:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: Active manager daemon smithi026.tawexg restarted 2024-04-16T07:12:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: Activating manager daemon smithi026.tawexg 2024-04-16T07:12:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: osdmap e3: 0 total, 0 up, 0 in 2024-04-16T07:12:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: mgrmap e10: smithi026.tawexg(active, starting, since 0.00932579s) 2024-04-16T07:12:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:12:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:12:01.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:12:01.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:12:01.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:12:01.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:12:01.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:00 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:12:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:01 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:12:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:01 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:12:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:01 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:12:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:01 smithi026 ceph-mon[28805]: mgrmap e11: smithi026.tawexg(active, since 1.01063s) 2024-04-16T07:12:02.379 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-16T07:12:02.379 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-16T07:12:02.379 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T07:12:02.379 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-16T07:12:02.380 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.958+0000 7f7cb9c00700 1 Processor -- start 2024-04-16T07:12:02.380 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.958+0000 7f7cb9c00700 1 -- start start 2024-04-16T07:12:02.380 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.959+0000 7f7cb9c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:02.380 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.959+0000 7f7cb9c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4103250 con 0x7f7cb4102900 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.959+0000 7f7cb2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.959+0000 7f7cb2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53726/0 (socket says 172.21.15.26:53726) 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.959+0000 7f7cb2a00700 1 -- 172.21.15.26:0/1989834497 learned_addr learned my addr 172.21.15.26:0/1989834497 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.960+0000 7f7cb2a00700 1 -- 172.21.15.26:0/1989834497 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4103390 con 0x7f7cb4102900 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.960+0000 7f7cb2a00700 1 --2- 172.21.15.26:0/1989834497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40098e0 tx=0x7f7ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794ca046462c2c8c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.960+0000 7f7cb1600700 1 -- 172.21.15.26:0/1989834497 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca400b860 con 0x7f7cb4102900 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.960+0000 7f7cb1600700 1 -- 172.21.15.26:0/1989834497 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca400be90 con 0x7f7cb4102900 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.960+0000 7f7cb1600700 1 -- 172.21.15.26:0/1989834497 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca40103c0 con 0x7f7cb4102900 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1989834497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 msgr2=0x7f7cb4102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1989834497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40098e0 tx=0x7f7ca4009250 comp rx=0 tx=0).stop 2024-04-16T07:12:02.384 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1989834497 shutdown_connections 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1989834497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4102d10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1989834497 >> 172.21.15.26:0/1989834497 conn(0x7f7cb40fe0a0 msgr2=0x7f7cb41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1989834497 shutdown_connections 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.961+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1989834497 wait complete. 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.962+0000 7f7cb9c00700 1 Processor -- start 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.962+0000 7f7cb9c00700 1 -- start start 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.962+0000 7f7cb9c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.962+0000 7f7cb9c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb40780a0 con 0x7f7cb4102900 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.963+0000 7f7cb2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.963+0000 7f7cb2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53730/0 (socket says 172.21.15.26:53730) 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.963+0000 7f7cb2a00700 1 -- 172.21.15.26:0/1903523160 learned_addr learned my addr 172.21.15.26:0/1903523160 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:02.385 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.963+0000 7f7cb2a00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40094f0 con 0x7f7cb4102900 2024-04-16T07:12:02.386 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.963+0000 7f7cb2a00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006b00 tx=0x7f7ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:02.386 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.964+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca4010840 con 0x7f7cb4102900 2024-04-16T07:12:02.386 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.964+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca4010e70 con 0x7f7cb4102900 2024-04-16T07:12:02.390 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.964+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca401ae70 con 0x7f7cb4102900 2024-04-16T07:12:02.390 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.964+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb4076110 con 0x7f7cb4102900 2024-04-16T07:12:02.390 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.964+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb4076600 con 0x7f7cb4102900 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.965+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f7ca401a780 con 0x7f7cb4102900 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.966+0000 7f7caaa00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.966+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c980394b0 con 0x7f7c980368f0 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.966+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7ca404ac40 con 0x7f7cb4102900 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.966+0000 7f7cb2000700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:54.966+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:55.167+0000 7f7cb2000700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:55.167+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:55.568+0000 7f7cb2000700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:55.568+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:56.369+0000 7f7cb2000700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:56.369+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:12:02.391 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:57.971+0000 7f7cb2000700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3603264388 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:11:57.971+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:00.583+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 10) v1 ==== 43156+0+0 (secure 0 0 0) 0x7f7ca401a430 con 0x7f7cb4102900 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:00.583+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 msgr2=0x7f7c98038da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:00.583+0000 7f7caaa00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.585+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 11) v1 ==== 43283+0+0 (secure 0 0 0) 0x7f7ca40109a0 con 0x7f7cb4102900 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.585+0000 7f7caaa00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 0x7f7c9803c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.585+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c980394b0 con 0x7f7c9803a3a0 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.587+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 0x7f7c9803c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.588+0000 7f7cb2000700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 0x7f7c9803c780 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00039d0 tx=0x7f7ca0009350 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.588+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f7c980394b0 con 0x7f7c9803a3a0 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.595+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7cb404bed0 con 0x7f7c9803a3a0 2024-04-16T07:12:02.392 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7caaa00700 1 -- 172.21.15.26:0/1903523160 <== mgr.14158 v2:172.21.15.26:6800/2140801142 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7cb404bed0 con 0x7f7c9803a3a0 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 msgr2=0x7f7c9803c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 0x7f7c9803c780 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00039d0 tx=0x7f7ca0009350 comp rx=0 tx=0).stop 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 msgr2=0x7f7cb4077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006b00 tx=0x7f7ca4009250 comp rx=0 tx=0).stop 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 shutdown_connections 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7c9803a3a0 0x7f7c9803c780 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7cb4102900 0x7f7cb4077b60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 --2- 172.21.15.26:0/1903523160 >> [v2:172.21.15.26:6800/3603264388,v1:172.21.15.26:6801/3603264388] conn(0x7f7c980368f0 0x7f7c98038da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.596+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 >> 172.21.15.26:0/1903523160 conn(0x7f7cb40fe0a0 msgr2=0x7f7cb40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.597+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 shutdown_connections 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:01.597+0000 7f7cb9c00700 1 -- 172.21.15.26:0/1903523160 wait complete. 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 9 is available 2024-04-16T07:12:02.393 INFO:teuthology.orchestra.run.smithi026.stdout:Generating a dashboard self-signed certificate... 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: [16/Apr/2024:07:12:01] ENGINE Bus STARTING 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: [16/Apr/2024:07:12:01] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: [16/Apr/2024:07:12:01] ENGINE Bus STARTED 2024-04-16T07:12:02.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:02 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:04.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:03 smithi026 ceph-mon[28805]: mgrmap e12: smithi026.tawexg(active, since 2s) 2024-04-16T07:12:04.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:03 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:04.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-16T07:12:04.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.433+0000 7f46e1000700 1 Processor -- start 2024-04-16T07:12:04.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.434+0000 7f46e1000700 1 -- start start 2024-04-16T07:12:04.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.434+0000 7f46e1000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:04.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.434+0000 7f46e1000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc077fb0 con 0x7f46dc077ba0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.435+0000 7f46d9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.435+0000 7f46d9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37548/0 (socket says 172.21.15.26:37548) 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.435+0000 7f46d9e00700 1 -- 172.21.15.26:0/1564808384 learned_addr learned my addr 172.21.15.26:0/1564808384 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.435+0000 7f46d9e00700 1 -- 172.21.15.26:0/1564808384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc076690 con 0x7f46dc077ba0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.436+0000 7f46d9e00700 1 --2- 172.21.15.26:0/1564808384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0098e0 tx=0x7f46cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa542633ebe379f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.436+0000 7f46d8a00700 1 -- 172.21.15.26:0/1564808384 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc00b860 con 0x7f46dc077ba0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.436+0000 7f46d8a00700 1 -- 172.21.15.26:0/1564808384 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46cc00be90 con 0x7f46dc077ba0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.436+0000 7f46d8a00700 1 -- 172.21.15.26:0/1564808384 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc0103c0 con 0x7f46dc077ba0 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 -- 172.21.15.26:0/1564808384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 msgr2=0x7f46dc076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 --2- 172.21.15.26:0/1564808384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0098e0 tx=0x7f46cc009250 comp rx=0 tx=0).stop 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 -- 172.21.15.26:0/1564808384 shutdown_connections 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 --2- 172.21.15.26:0/1564808384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc076150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 -- 172.21.15.26:0/1564808384 >> 172.21.15.26:0/1564808384 conn(0x7f46dc0fe0a0 msgr2=0x7f46dc1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 -- 172.21.15.26:0/1564808384 shutdown_connections 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.437+0000 7f46e1000700 1 -- 172.21.15.26:0/1564808384 wait complete. 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.438+0000 7f46e1000700 1 Processor -- start 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.438+0000 7f46e1000700 1 -- start start 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.439+0000 7f46e1000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.439+0000 7f46e1000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc113dc0 con 0x7f46dc077ba0 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.439+0000 7f46d9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.439+0000 7f46d9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37552/0 (socket says 172.21.15.26:37552) 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.439+0000 7f46d9e00700 1 -- 172.21.15.26:0/1261291819 learned_addr learned my addr 172.21.15.26:0/1261291819 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46d9e00700 1 -- 172.21.15.26:0/1261291819 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0094f0 con 0x7f46dc077ba0 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46d9e00700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0105b0 tx=0x7f46cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc010840 con 0x7f46dc077ba0 2024-04-16T07:12:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46cc010e70 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc01ae30 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.440+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc113fc0 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.441+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc114b30 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.442+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc05f5f0 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.442+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f46cc01a430 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.443+0000 7f46d2000700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 0x7f46c0038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.443+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f46cc04a3f0 con 0x7f46dc077ba0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.443+0000 7f46d9400700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 0x7f46c0038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:04.458 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.444+0000 7f46d9400700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 0x7f46c0038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009990 tx=0x7f46c8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.448+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f46cc02b4d0 con 0x7f46dc077ba0 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.673+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f46dc0ff230 con 0x7f46c00368a0 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.718+0000 7f46d2000700 1 -- 172.21.15.26:0/1261291819 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f46dc0ff230 con 0x7f46c00368a0 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.721+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 msgr2=0x7f46c0038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.721+0000 7f46e1000700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 0x7f46c0038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009990 tx=0x7f46c8006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.721+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 msgr2=0x7f46dc113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.721+0000 7f46e1000700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0105b0 tx=0x7f46cc009250 comp rx=0 tx=0).stop 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 shutdown_connections 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f46c00368a0 0x7f46c0038d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 --2- 172.21.15.26:0/1261291819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f46dc077ba0 0x7f46dc113880 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 >> 172.21.15.26:0/1261291819 conn(0x7f46dc0fe0a0 msgr2=0x7f46dc0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 shutdown_connections 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:03.722+0000 7f46e1000700 1 -- 172.21.15.26:0/1261291819 wait complete. 2024-04-16T07:12:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial admin user... 2024-04-16T07:12:04.982 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:04 smithi026 ceph-mon[28805]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:12:04.983 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:04.983 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:06.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:05 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:06.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:05 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:07.007 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:06 smithi026 ceph-mon[28805]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:12:07.189 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$TUNbk2ZSHmXmgcc0LOxl1ew1yIre/2q8M6HegM2LLAxb/jrek3kgu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713251525, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-16T07:12:07.189 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.047+0000 7fdd8f800700 1 Processor -- start 2024-04-16T07:12:07.189 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.047+0000 7fdd8f800700 1 -- start start 2024-04-16T07:12:07.189 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.048+0000 7fdd8f800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:07.189 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.048+0000 7fdd8f800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd881031e0 con 0x7fdd88102890 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.048+0000 7fdd8ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.048+0000 7fdd8ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37554/0 (socket says 172.21.15.26:37554) 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.048+0000 7fdd8ce00700 1 -- 172.21.15.26:0/2476459578 learned_addr learned my addr 172.21.15.26:0/2476459578 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.049+0000 7fdd8ce00700 1 -- 172.21.15.26:0/2476459578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88103320 con 0x7fdd88102890 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.049+0000 7fdd8ce00700 1 --2- 172.21.15.26:0/2476459578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0098e0 tx=0x7fdd7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dfebc97464fa08 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.049+0000 7fdd87400700 1 -- 172.21.15.26:0/2476459578 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c00b860 con 0x7fdd88102890 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.049+0000 7fdd87400700 1 -- 172.21.15.26:0/2476459578 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd7c00be90 con 0x7fdd88102890 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.049+0000 7fdd87400700 1 -- 172.21.15.26:0/2476459578 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c0103c0 con 0x7fdd88102890 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 -- 172.21.15.26:0/2476459578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 msgr2=0x7fdd88102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 --2- 172.21.15.26:0/2476459578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0098e0 tx=0x7fdd7c009250 comp rx=0 tx=0).stop 2024-04-16T07:12:07.190 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 -- 172.21.15.26:0/2476459578 shutdown_connections 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 --2- 172.21.15.26:0/2476459578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd88102ca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 -- 172.21.15.26:0/2476459578 >> 172.21.15.26:0/2476459578 conn(0x7fdd880fe050 msgr2=0x7fdd88100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 -- 172.21.15.26:0/2476459578 shutdown_connections 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.050+0000 7fdd8f800700 1 -- 172.21.15.26:0/2476459578 wait complete. 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.051+0000 7fdd8f800700 1 Processor -- start 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.051+0000 7fdd8f800700 1 -- start start 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8f800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8f800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88113d30 con 0x7fdd88102890 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37564/0 (socket says 172.21.15.26:37564) 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8ce00700 1 -- 172.21.15.26:0/3825984363 learned_addr learned my addr 172.21.15.26:0/3825984363 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.052+0000 7fdd8ce00700 1 -- 172.21.15.26:0/3825984363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0094f0 con 0x7fdd88102890 2024-04-16T07:12:07.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd8ce00700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0090b0 tx=0x7fdd7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c010840 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd7c010e70 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd88113f30 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c024e40 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.053+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd88114aa0 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.054+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7fdd7c01a4c0 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.054+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8805f5f0 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.055+0000 7fdd84c00700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 0x7fdd70038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.055+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdd7c04a8d0 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.055+0000 7fdd87e00700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 0x7fdd70038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.056+0000 7fdd87e00700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 0x7fdd70038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009990 tx=0x7fdd74006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.060+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdd7c018940 con 0x7fdd88102890 2024-04-16T07:12:07.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.299+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdd8805f410 con 0x7fdd700368a0 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.559+0000 7fdd84c00700 1 -- 172.21.15.26:0/3825984363 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fdd8805f410 con 0x7fdd700368a0 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 msgr2=0x7fdd70038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 0x7fdd70038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009990 tx=0x7fdd74006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 msgr2=0x7fdd881137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0090b0 tx=0x7fdd7c009250 comp rx=0 tx=0).stop 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 shutdown_connections 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fdd700368a0 0x7fdd70038d50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 --2- 172.21.15.26:0/3825984363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd88102890 0x7fdd881137f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 >> 172.21.15.26:0/3825984363 conn(0x7fdd880fe050 msgr2=0x7fdd88077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 shutdown_connections 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:05.563+0000 7fdd8f800700 1 -- 172.21.15.26:0/3825984363 wait complete. 2024-04-16T07:12:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:Fetching dashboard port number... 2024-04-16T07:12:08.235 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:07 smithi026 ceph-mon[28805]: mgrmap e13: smithi026.tawexg(active, since 6s) 2024-04-16T07:12:08.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 8443 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.006+0000 7feff4200700 1 Processor -- start 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.006+0000 7feff4200700 1 -- start start 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.007+0000 7feff4200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.007+0000 7feff4200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec07d450 con 0x7fefec07cfb0 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.007+0000 7feff1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.007+0000 7feff1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37216/0 (socket says 172.21.15.26:37216) 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.007+0000 7feff1800700 1 -- 172.21.15.26:0/2072908283 learned_addr learned my addr 172.21.15.26:0/2072908283 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.008+0000 7feff1800700 1 -- 172.21.15.26:0/2072908283 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec07d590 con 0x7fefec07cfb0 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.008+0000 7feff1800700 1 --2- 172.21.15.26:0/2072908283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fefe00098e0 tx=0x7fefe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1bfe24af24742bb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.008+0000 7fefebe00700 1 -- 172.21.15.26:0/2072908283 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe000b860 con 0x7fefec07cfb0 2024-04-16T07:12:08.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.008+0000 7fefebe00700 1 -- 172.21.15.26:0/2072908283 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe000be90 con 0x7fefec07cfb0 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7fefebe00700 1 -- 172.21.15.26:0/2072908283 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe00103c0 con 0x7fefec07cfb0 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7feff4200700 1 -- 172.21.15.26:0/2072908283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 msgr2=0x7fefec10bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7feff4200700 1 --2- 172.21.15.26:0/2072908283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fefe00098e0 tx=0x7fefe0009250 comp rx=0 tx=0).stop 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7feff4200700 1 -- 172.21.15.26:0/2072908283 shutdown_connections 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7feff4200700 1 --2- 172.21.15.26:0/2072908283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec07cfb0 0x7fefec10bd00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.009+0000 7feff4200700 1 -- 172.21.15.26:0/2072908283 >> 172.21.15.26:0/2072908283 conn(0x7fefec078560 msgr2=0x7fefec07a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.010+0000 7feff4200700 1 -- 172.21.15.26:0/2072908283 shutdown_connections 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.010+0000 7feff4200700 1 -- 172.21.15.26:0/2072908283 wait complete. 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.011+0000 7feff4200700 1 Processor -- start 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.011+0000 7feff4200700 1 -- start start 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff4200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:08.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff4200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec118750 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37226/0 (socket says 172.21.15.26:37226) 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff1800700 1 -- 172.21.15.26:0/1031716751 learned_addr learned my addr 172.21.15.26:0/1031716751 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.012+0000 7feff1800700 1 -- 172.21.15.26:0/1031716751 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00094f0 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7feff1800700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fefe0011d50 tx=0x7fefe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe0011910 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe001f070 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe001a9c0 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec118fb0 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.013+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec119440 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.014+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fefe001ab20 con 0x7fefec117e00 2024-04-16T07:12:08.919 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.014+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec04bed0 con 0x7fefec117e00 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.015+0000 7fefe9600700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 0x7fefd4038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.015+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fefe0021070 con 0x7fefec117e00 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.015+0000 7feff0e00700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 0x7fefd4038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.016+0000 7feff0e00700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 0x7fefd4038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009990 tx=0x7fefd8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.020+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fefe0024560 con 0x7fefec117e00 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.236+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fefec1188f0 con 0x7fefec117e00 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.237+0000 7fefe9600700 1 -- 172.21.15.26:0/1031716751 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fefe0024380 con 0x7fefec117e00 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 msgr2=0x7fefd4038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 0x7fefd4038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009990 tx=0x7fefd8006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 msgr2=0x7fefec118210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fefe0011d50 tx=0x7fefe0009250 comp rx=0 tx=0).stop 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 shutdown_connections 2024-04-16T07:12:08.920 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fefd4036990 0x7fefd4038e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:08.921 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 --2- 172.21.15.26:0/1031716751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefec117e00 0x7fefec118210 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:08.921 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 >> 172.21.15.26:0/1031716751 conn(0x7fefec078560 msgr2=0x7fefec07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:08.921 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.241+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 shutdown_connections 2024-04-16T07:12:08.921 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:08.242+0000 7feff4200700 1 -- 172.21.15.26:0/1031716751 wait complete. 2024-04-16T07:12:08.951 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T07:12:08.951 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-16T07:12:08.978 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T07:12:08.978 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-16T07:12:08.978 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-16T07:12:08.978 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph Dashboard is now available at: 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout: URL: https://smithi026.front.sepia.ceph.com:8443/ 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout: User: admin 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout: Password: ga9hoy7ipv 2024-04-16T07:12:08.986 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:08.987 INFO:teuthology.orchestra.run.smithi026.stdout:Saving cluster configuration to /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config directory 2024-04-16T07:12:08.989 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling autotune for osd_memory_target 2024-04-16T07:12:09.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:08 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1031716751' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-16T07:12:10.568 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.614+0000 7f1cfbe00700 1 Processor -- start 2024-04-16T07:12:10.568 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.614+0000 7f1cfbe00700 1 -- start start 2024-04-16T07:12:10.568 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.615+0000 7f1cfbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:10.568 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.615+0000 7f1cfbe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4105620 con 0x7f1cf4104c20 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.615+0000 7f1cf9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.615+0000 7f1cf9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37228/0 (socket says 172.21.15.26:37228) 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.615+0000 7f1cf9400700 1 -- 172.21.15.26:0/160521575 learned_addr learned my addr 172.21.15.26:0/160521575 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.616+0000 7f1cf9400700 1 -- 172.21.15.26:0/160521575 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf4105760 con 0x7f1cf4104c20 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.616+0000 7f1cf9400700 1 --2- 172.21.15.26:0/160521575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba7ec0c8c555131 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.616+0000 7f1cf3e00700 1 -- 172.21.15.26:0/160521575 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce800b860 con 0x7f1cf4104c20 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cf3e00700 1 -- 172.21.15.26:0/160521575 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ce800be90 con 0x7f1cf4104c20 2024-04-16T07:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cf3e00700 1 -- 172.21.15.26:0/160521575 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce80103c0 con 0x7f1cf4104c20 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 -- 172.21.15.26:0/160521575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 msgr2=0x7f1cf4105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/160521575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8009250 comp rx=0 tx=0).stop 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 -- 172.21.15.26:0/160521575 shutdown_connections 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/160521575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4105050 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 -- 172.21.15.26:0/160521575 >> 172.21.15.26:0/160521575 conn(0x7f1cf4078560 msgr2=0x7f1cf4078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.617+0000 7f1cfbe00700 1 -- 172.21.15.26:0/160521575 shutdown_connections 2024-04-16T07:12:10.570 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.618+0000 7f1cfbe00700 1 -- 172.21.15.26:0/160521575 wait complete. 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.618+0000 7f1cfbe00700 1 Processor -- start 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cfbe00700 1 -- start start 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cfbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cfbe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4113d50 con 0x7f1cf4104c20 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cf9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cf9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37230/0 (socket says 172.21.15.26:37230) 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.619+0000 7f1cf9400700 1 -- 172.21.15.26:0/1233213978 learned_addr learned my addr 172.21.15.26:0/1233213978 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cf9400700 1 -- 172.21.15.26:0/1233213978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80094f0 con 0x7f1cf4104c20 2024-04-16T07:12:10.571 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cf9400700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80105b0 tx=0x7f1ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce8010840 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ce8010e70 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf4113f50 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.620+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce80243f0 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.621+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf4114ac0 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.622+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f1ce801a7c0 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.622+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf405f5f0 con 0x7f1cf4104c20 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.623+0000 7f1cf1600700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 0x7f1cdc038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:10.572 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.623+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1ce804a7c0 con 0x7f1cf4104c20 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.623+0000 7f1cf8a00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 0x7f1cdc038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.624+0000 7f1cf8a00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 0x7f1cdc038e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009990 tx=0x7f1ce0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.628+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1ce8018bd0 con 0x7f1cf4104c20 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.841+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f1cf4105760 con 0x7f1cf4104c20 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.841+0000 7f1cf1600700 1 -- 172.21.15.26:0/1233213978 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1cf4105760 con 0x7f1cf4104c20 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.844+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 msgr2=0x7f1cdc038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.844+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 0x7f1cdc038e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009990 tx=0x7f1ce0006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:10.573 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.844+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 msgr2=0x7f1cf4113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.844+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80105b0 tx=0x7f1ce8009250 comp rx=0 tx=0).stop 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 shutdown_connections 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f1cdc036990 0x7f1cdc038e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 --2- 172.21.15.26:0/1233213978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cf4104c20 0x7f1cf4113810 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 >> 172.21.15.26:0/1233213978 conn(0x7f1cf4078560 msgr2=0x7f1cf41023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 shutdown_connections 2024-04-16T07:12:10.574 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:09.845+0000 7f1cfbe00700 1 -- 172.21.15.26:0/1233213978 wait complete. 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.224+0000 7fbe09c00700 1 Processor -- start 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.224+0000 7fbe09c00700 1 -- start start 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.224+0000 7fbe09c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.224+0000 7fbe09c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe04077f60 con 0x7fbe04077b50 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.225+0000 7fbe02a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.225+0000 7fbe02a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37246/0 (socket says 172.21.15.26:37246) 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.225+0000 7fbe02a00700 1 -- 172.21.15.26:0/2701743040 learned_addr learned my addr 172.21.15.26:0/2701743040 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.226+0000 7fbe02a00700 1 -- 172.21.15.26:0/2701743040 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04076690 con 0x7fbe04077b50 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.226+0000 7fbe02a00700 1 --2- 172.21.15.26:0/2701743040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40098e0 tx=0x7fbdf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86e6cac318a27ad server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.226+0000 7fbe01600700 1 -- 172.21.15.26:0/2701743040 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf400b860 con 0x7fbe04077b50 2024-04-16T07:12:12.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.226+0000 7fbe01600700 1 -- 172.21.15.26:0/2701743040 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdf400be90 con 0x7fbe04077b50 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.226+0000 7fbe01600700 1 -- 172.21.15.26:0/2701743040 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf40103c0 con 0x7fbe04077b50 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 -- 172.21.15.26:0/2701743040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 msgr2=0x7fbe04076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 --2- 172.21.15.26:0/2701743040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40098e0 tx=0x7fbdf4009250 comp rx=0 tx=0).stop 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 -- 172.21.15.26:0/2701743040 shutdown_connections 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 --2- 172.21.15.26:0/2701743040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe04076150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 -- 172.21.15.26:0/2701743040 >> 172.21.15.26:0/2701743040 conn(0x7fbe040fe110 msgr2=0x7fbe04100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.227+0000 7fbe09c00700 1 -- 172.21.15.26:0/2701743040 shutdown_connections 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.228+0000 7fbe09c00700 1 -- 172.21.15.26:0/2701743040 wait complete. 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.228+0000 7fbe09c00700 1 Processor -- start 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe09c00700 1 -- start start 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe09c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe09c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe04113de0 con 0x7fbe04077b50 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe02a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe02a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37252/0 (socket says 172.21.15.26:37252) 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.229+0000 7fbe02a00700 1 -- 172.21.15.26:0/4162148210 learned_addr learned my addr 172.21.15.26:0/4162148210 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbe02a00700 1 -- 172.21.15.26:0/4162148210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40094f0 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbe02a00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4011c10 tx=0x7fbdf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4010630 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdf4010c60 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf401ad10 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.230+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe04113fe0 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.231+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe04114b50 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.232+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0405f5f0 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.232+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fbdf401a5f0 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.232+0000 7fbdfaa00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 0x7fbde8038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.232+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbdf404a1a0 con 0x7fbe04077b50 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.233+0000 7fbe02000700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 0x7fbde8038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.233+0000 7fbe02000700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 0x7fbde8038e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009990 tx=0x7fbdf0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:12.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.238+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbdf40249a0 con 0x7fbe04077b50 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.575+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fbe0404bed0 con 0x7fbe04077b50 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.579+0000 7fbdfaa00700 1 -- 172.21.15.26:0/4162148210 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fbdf4020030 con 0x7fbe04077b50 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.582+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 msgr2=0x7fbde8038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.582+0000 7fbe09c00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 0x7fbde8038e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009990 tx=0x7fbdf0006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.582+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 msgr2=0x7fbe041138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.582+0000 7fbe09c00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4011c10 tx=0x7fbdf4009250 comp rx=0 tx=0).stop 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 shutdown_connections 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbde8036990 0x7fbde8038e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 --2- 172.21.15.26:0/4162148210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe04077b50 0x7fbe041138a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 >> 172.21.15.26:0/4162148210 conn(0x7fbe040fe110 msgr2=0x7fbe040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 shutdown_connections 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-16T07:12:11.583+0000 7fbe09c00700 1 -- 172.21.15.26:0/4162148210 wait complete. 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:Or, if you are only running a single cluster on this host: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: ceph telemetry on 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:For more information see: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:Bootstrap complete. 2024-04-16T07:12:12.367 INFO:tasks.cephadm:Fetching config... 2024-04-16T07:12:12.368 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:12:12.368 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-16T07:12:12.416 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-16T07:12:12.416 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:12:12.416 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-16T07:12:12.495 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-16T07:12:12.495 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:12:12.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/keyring of=/dev/stdout 2024-04-16T07:12:12.623 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-16T07:12:12.623 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:12:12.623 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-16T07:12:12.719 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-16T07:12:12.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWPFNAM+fKiKz6j2R3V81/tafo91vupp0KnQJh4J9h+QjqY0Hofp4Cx9mSI6+ift7FAkj2K4wk+ab2IEmTaCE6HPKh0nX1QkJ4gc3Za/x3cpmsiFoNR66t1TWTF/ggJHomhF7vKK3B1eUENmVN3g6PC3GshqaCV8eDnrLEd3v2IxiLJxiVMmUV8mixy72TinCj6fMG2/LUxpzp3jtqSRvQxnAk6P1rsPE2vKn65yvjmd7POD2tkkNigs9U7r/meluij6pDR5wy9Iqol9jtHTq2NK9vPRLgRrQWq1pvmbq4bb3djbnPIMaMwnBTBvA0FE992anAqiJYANKFcX1RLhHpVxsADTm/xixGKcMkazyGOQr0MtLYy//RnF8AAQ/vvjyTjnGV1XCk9Ch7GdKjEebbUMH8iQk3nasbE2MgX7XFzus5SbHewWElS3WT0UxjLLt8ujMox00v9M2S/bS84s4BZL/nq08n3Pq470I5S4oAdr+dVMD28mR7+39xAf0XFdk= ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T07:12:12.785 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:12 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4162148210' entity='client.admin' 2024-04-16T07:12:12.836 INFO:teuthology.orchestra.run.smithi026.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWPFNAM+fKiKz6j2R3V81/tafo91vupp0KnQJh4J9h+QjqY0Hofp4Cx9mSI6+ift7FAkj2K4wk+ab2IEmTaCE6HPKh0nX1QkJ4gc3Za/x3cpmsiFoNR66t1TWTF/ggJHomhF7vKK3B1eUENmVN3g6PC3GshqaCV8eDnrLEd3v2IxiLJxiVMmUV8mixy72TinCj6fMG2/LUxpzp3jtqSRvQxnAk6P1rsPE2vKn65yvjmd7POD2tkkNigs9U7r/meluij6pDR5wy9Iqol9jtHTq2NK9vPRLgRrQWq1pvmbq4bb3djbnPIMaMwnBTBvA0FE992anAqiJYANKFcX1RLhHpVxsADTm/xixGKcMkazyGOQr0MtLYy//RnF8AAQ/vvjyTjnGV1XCk9Ch7GdKjEebbUMH8iQk3nasbE2MgX7XFzus5SbHewWElS3WT0UxjLLt8ujMox00v9M2S/bS84s4BZL/nq08n3Pq470I5S4oAdr+dVMD28mR7+39xAf0XFdk= ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:12:12.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWPFNAM+fKiKz6j2R3V81/tafo91vupp0KnQJh4J9h+QjqY0Hofp4Cx9mSI6+ift7FAkj2K4wk+ab2IEmTaCE6HPKh0nX1QkJ4gc3Za/x3cpmsiFoNR66t1TWTF/ggJHomhF7vKK3B1eUENmVN3g6PC3GshqaCV8eDnrLEd3v2IxiLJxiVMmUV8mixy72TinCj6fMG2/LUxpzp3jtqSRvQxnAk6P1rsPE2vKn65yvjmd7POD2tkkNigs9U7r/meluij6pDR5wy9Iqol9jtHTq2NK9vPRLgRrQWq1pvmbq4bb3djbnPIMaMwnBTBvA0FE992anAqiJYANKFcX1RLhHpVxsADTm/xixGKcMkazyGOQr0MtLYy//RnF8AAQ/vvjyTjnGV1XCk9Ch7GdKjEebbUMH8iQk3nasbE2MgX7XFzus5SbHewWElS3WT0UxjLLt8ujMox00v9M2S/bS84s4BZL/nq08n3Pq470I5S4oAdr+dVMD28mR7+39xAf0XFdk= ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T07:12:13.027 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWPFNAM+fKiKz6j2R3V81/tafo91vupp0KnQJh4J9h+QjqY0Hofp4Cx9mSI6+ift7FAkj2K4wk+ab2IEmTaCE6HPKh0nX1QkJ4gc3Za/x3cpmsiFoNR66t1TWTF/ggJHomhF7vKK3B1eUENmVN3g6PC3GshqaCV8eDnrLEd3v2IxiLJxiVMmUV8mixy72TinCj6fMG2/LUxpzp3jtqSRvQxnAk6P1rsPE2vKn65yvjmd7POD2tkkNigs9U7r/meluij6pDR5wy9Iqol9jtHTq2NK9vPRLgRrQWq1pvmbq4bb3djbnPIMaMwnBTBvA0FE992anAqiJYANKFcX1RLhHpVxsADTm/xixGKcMkazyGOQr0MtLYy//RnF8AAQ/vvjyTjnGV1XCk9Ch7GdKjEebbUMH8iQk3nasbE2MgX7XFzus5SbHewWElS3WT0UxjLLt8ujMox00v9M2S/bS84s4BZL/nq08n3Pq470I5S4oAdr+dVMD28mR7+39xAf0XFdk= ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:12:13.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-16T07:12:13.547 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:12:16.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:16 smithi026 ceph-mon[28805]: Deploying daemon crash.smithi026 on smithi026 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 -- 172.21.15.26:0/1099783892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 msgr2=0x7fb6140fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 --2- 172.21.15.26:0/1099783892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb6140fce80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb6040048c0 tx=0x7fb604009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 -- 172.21.15.26:0/1099783892 shutdown_connections 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 --2- 172.21.15.26:0/1099783892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb6140fce80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 -- 172.21.15.26:0/1099783892 >> 172.21.15.26:0/1099783892 conn(0x7fb6140f81d0 msgr2=0x7fb6140fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 -- 172.21.15.26:0/1099783892 shutdown_connections 2024-04-16T07:12:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.425+0000 7fb618a00700 1 -- 172.21.15.26:0/1099783892 wait complete. 2024-04-16T07:12:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.426+0000 7fb618a00700 1 Processor -- start 2024-04-16T07:12:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.426+0000 7fb618a00700 1 -- start start 2024-04-16T07:12:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb618a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb618a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61410f7f0 con 0x7fb6140fca70 2024-04-16T07:12:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb611e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb611e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37270/0 (socket says 172.21.15.26:37270) 2024-04-16T07:12:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb611e00700 1 -- 172.21.15.26:0/2752088386 learned_addr learned my addr 172.21.15.26:0/2752088386 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.427+0000 7fb611e00700 1 -- 172.21.15.26:0/2752088386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb604005040 con 0x7fb6140fca70 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb611e00700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb604009680 tx=0x7fb604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60400b5c0 con 0x7fb6140fca70 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb60400be70 con 0x7fb6140fca70 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6040135a0 con 0x7fb6140fca70 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61410f9f0 con 0x7fb6140fca70 2024-04-16T07:12:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.428+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb614110560 con 0x7fb6140fca70 2024-04-16T07:12:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.430+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb614102f80 con 0x7fb6140fca70 2024-04-16T07:12:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.430+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb60401ca90 con 0x7fb6140fca70 2024-04-16T07:12:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.430+0000 7fb60a000700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 0x7fb5f8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.430+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb6040493d0 con 0x7fb6140fca70 2024-04-16T07:12:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.431+0000 7fb611400700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 0x7fb5f8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.432+0000 7fb611400700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 0x7fb5f8038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb600009990 tx=0x7fb600006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:16.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.436+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb604011940 con 0x7fb6140fca70 2024-04-16T07:12:16.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.653+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb614110860 con 0x7fb6140fca70 2024-04-16T07:12:16.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.657+0000 7fb60a000700 1 -- 172.21.15.26:0/2752088386 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb604021020 con 0x7fb6140fca70 2024-04-16T07:12:16.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.660+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 msgr2=0x7fb5f8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:16.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.660+0000 7fb618a00700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 0x7fb5f8038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb600009990 tx=0x7fb600006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:16.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 msgr2=0x7fb61410f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:16.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb604009680 tx=0x7fb604009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 shutdown_connections 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb5f80368d0 0x7fb5f8038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 --2- 172.21.15.26:0/2752088386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6140fca70 0x7fb61410f2b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.661+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 >> 172.21.15.26:0/2752088386 conn(0x7fb6140f81d0 msgr2=0x7fb6140fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.662+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 shutdown_connections 2024-04-16T07:12:16.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:16.662+0000 7fb618a00700 1 -- 172.21.15.26:0/2752088386 wait complete. 2024-04-16T07:12:17.408 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-16T07:12:17.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-16T07:12:18.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:17 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2752088386' entity='client.admin' 2024-04-16T07:12:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 -- 172.21.15.26:0/2844468438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 msgr2=0x7f6fac103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2844468438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac103490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009c60 tx=0x7f6fa0004d10 comp rx=0 tx=0).stop 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 -- 172.21.15.26:0/2844468438 shutdown_connections 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2844468438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac103490 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 -- 172.21.15.26:0/2844468438 >> 172.21.15.26:0/2844468438 conn(0x7f6fac072300 msgr2=0x7f6fac074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 -- 172.21.15.26:0/2844468438 shutdown_connections 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.594+0000 7f6fb4600700 1 -- 172.21.15.26:0/2844468438 wait complete. 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.596+0000 7f6fb4600700 1 Processor -- start 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.596+0000 7f6fb4600700 1 -- start start 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.596+0000 7f6fb4600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.596+0000 7f6fb4600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac113c10 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.597+0000 7f6fb1c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.597+0000 7f6fb1c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55398/0 (socket says 172.21.15.26:55398) 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.597+0000 7f6fb1c00700 1 -- 172.21.15.26:0/2747395329 learned_addr learned my addr 172.21.15.26:0/2747395329 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.597+0000 7f6fb1c00700 1 -- 172.21.15.26:0/2747395329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa0009940 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.597+0000 7f6fb1c00700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0004e60 tx=0x7f6fa0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.598+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa0012410 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.598+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fa0012a40 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.598+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac113e10 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.598+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa001ba40 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.598+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac114980 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.599+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f6fa0019040 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.599+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac109570 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.600+0000 7f6fa9600700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 0x7f6f94038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.600+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6fa0049cb0 con 0x7f6fac103080 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.600+0000 7f6fb1200700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 0x7f6f94038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.601+0000 7f6fb1200700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 0x7f6f94038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009990 tx=0x7f6f98006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.605+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6fa00227d0 con 0x7f6fac103080 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.829+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fac114bc0 con 0x7f6f940368d0 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.839+0000 7f6fa9600700 1 -- 172.21.15.26:0/2747395329 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fac114bc0 con 0x7f6f940368d0 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.841+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 msgr2=0x7f6f94038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.841+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 0x7f6f94038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009990 tx=0x7f6f98006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.841+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 msgr2=0x7f6fac1136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.841+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0004e60 tx=0x7f6fa0004d10 comp rx=0 tx=0).stop 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 shutdown_connections 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f6f940368d0 0x7f6f94038d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 --2- 172.21.15.26:0/2747395329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fac103080 0x7f6fac1136d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 >> 172.21.15.26:0/2747395329 conn(0x7f6fac072300 msgr2=0x7f6fac074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 shutdown_connections 2024-04-16T07:12:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:20.842+0000 7f6fb4600700 1 -- 172.21.15.26:0/2747395329 wait complete. 2024-04-16T07:12:21.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:21 smithi026 ceph-mon[28805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:21.895 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:21 smithi026 ceph-mon[28805]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:12:21.895 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:21 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:22.130 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-04-16T07:12:22.130 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:12:22.130 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-04-16T07:12:22.191 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:12:22.191 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:12:22.282 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-04-16T07:12:22.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch host add smithi104 2024-04-16T07:12:22.926 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:12:24.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:24.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:24.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:24.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:24.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:24.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:23 smithi026 ceph-mon[28805]: Deploying daemon node-exporter.smithi026 on smithi026 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 -- 172.21.15.26:0/2331864555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 msgr2=0x7f54480fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 --2- 172.21.15.26:0/2331864555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f54480fce60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f543c0048c0 tx=0x7f543c009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 -- 172.21.15.26:0/2331864555 shutdown_connections 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 --2- 172.21.15.26:0/2331864555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f54480fce60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 -- 172.21.15.26:0/2331864555 >> 172.21.15.26:0/2331864555 conn(0x7f54480f81d0 msgr2=0x7f54480fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 -- 172.21.15.26:0/2331864555 shutdown_connections 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.849+0000 7f5450800700 1 -- 172.21.15.26:0/2331864555 wait complete. 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.850+0000 7f5450800700 1 Processor -- start 2024-04-16T07:12:24.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.850+0000 7f5450800700 1 -- start start 2024-04-16T07:12:24.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.851+0000 7f5450800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:24.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.851+0000 7f5450800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54480722a0 con 0x7f54480fca50 2024-04-16T07:12:24.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.851+0000 7f544de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:24.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.851+0000 7f544de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55412/0 (socket says 172.21.15.26:55412) 2024-04-16T07:12:24.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.851+0000 7f544de00700 1 -- 172.21.15.26:0/2424802847 learned_addr learned my addr 172.21.15.26:0/2424802847 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f544de00700 1 -- 172.21.15.26:0/2424802847 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c005040 con 0x7f54480fca50 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f544de00700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f543c009680 tx=0x7f543c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f543c00b5c0 con 0x7f54480fca50 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f543c00be70 con 0x7f54480fca50 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448070370 con 0x7f54480fca50 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f543c0135a0 con 0x7f54480fca50 2024-04-16T07:12:24.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.852+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5448070860 con 0x7f54480fca50 2024-04-16T07:12:24.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.854+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544805f5f0 con 0x7f54480fca50 2024-04-16T07:12:24.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.854+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f543c004d30 con 0x7f54480fca50 2024-04-16T07:12:24.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.854+0000 7f5442000700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 0x7f5430038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:24.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.854+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f543c0493f0 con 0x7f54480fca50 2024-04-16T07:12:24.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.855+0000 7f544d400700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 0x7f5430038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:24.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.855+0000 7f544d400700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 0x7f5430038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5438009990 tx=0x7f5438006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:24.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:24.860+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f543c011780 con 0x7f54480fca50 2024-04-16T07:12:25.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:25.083+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7f54481064a0 con 0x7f5430036880 2024-04-16T07:12:26.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:25 smithi026 ceph-mon[28805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:26.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:25 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:27.134 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:26 smithi026 ceph-mon[28805]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:12:27.876 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:27 smithi026 ceph-mon[28805]: Deploying cephadm binary to smithi104 2024-04-16T07:12:27.877 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:27 smithi026 ceph-mon[28805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.495+0000 7f5442000700 1 -- 172.21.15.26:0/2424802847 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f54481064a0 con 0x7f5430036880 2024-04-16T07:12:29.496 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.498+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 msgr2=0x7f5430038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.498+0000 7f5450800700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 0x7f5430038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5438009990 tx=0x7f5438006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 msgr2=0x7f5448071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f543c009680 tx=0x7f543c009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 shutdown_connections 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f5430036880 0x7f5430038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 --2- 172.21.15.26:0/2424802847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54480fca50 0x7f5448071d60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 >> 172.21.15.26:0/2424802847 conn(0x7f54480f81d0 msgr2=0x7f5448103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 shutdown_connections 2024-04-16T07:12:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:29.499+0000 7f5450800700 1 -- 172.21.15.26:0/2424802847 wait complete. 2024-04-16T07:12:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: Deploying daemon alertmanager.smithi026 on smithi026 2024-04-16T07:12:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:29 smithi026 ceph-mon[28805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:30.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-16T07:12:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:30 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:30 smithi026 ceph-mon[28805]: Added host smithi104 2024-04-16T07:12:31.145 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:12:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:31 smithi026 ceph-mon[28805]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:31 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:34.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:33 smithi026 ceph-mon[28805]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:35.960 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:35 smithi026 ceph-mon[28805]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.394+0000 7f1984800700 1 -- 172.21.15.26:0/2401202634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 msgr2=0x7f197c0ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.394+0000 7f1984800700 1 --2- 172.21.15.26:0/2401202634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c0ff080 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f19700048c0 tx=0x7f1970009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.394+0000 7f1984800700 1 -- 172.21.15.26:0/2401202634 shutdown_connections 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.394+0000 7f1984800700 1 --2- 172.21.15.26:0/2401202634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c0ff080 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.394+0000 7f1984800700 1 -- 172.21.15.26:0/2401202634 >> 172.21.15.26:0/2401202634 conn(0x7f197c0fa5d0 msgr2=0x7f197c0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.395+0000 7f1984800700 1 -- 172.21.15.26:0/2401202634 shutdown_connections 2024-04-16T07:12:36.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.395+0000 7f1984800700 1 -- 172.21.15.26:0/2401202634 wait complete. 2024-04-16T07:12:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.395+0000 7f1984800700 1 Processor -- start 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1984800700 1 -- start start 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1984800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1984800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c10f830 con 0x7f197c0fec70 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1981e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1981e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39610/0 (socket says 172.21.15.26:39610) 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.396+0000 7f1981e00700 1 -- 172.21.15.26:0/1400838980 learned_addr learned my addr 172.21.15.26:0/1400838980 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1981e00700 1 -- 172.21.15.26:0/1400838980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970005040 con 0x7f197c0fec70 2024-04-16T07:12:36.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1981e00700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1970009680 tx=0x7f1970009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f197000b5c0 con 0x7f197c0fec70 2024-04-16T07:12:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f197000be70 con 0x7f197c0fec70 2024-04-16T07:12:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c10fa30 con 0x7f197c0fec70 2024-04-16T07:12:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.397+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19700135a0 con 0x7f197c0fec70 2024-04-16T07:12:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.398+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c110550 con 0x7f197c0fec70 2024-04-16T07:12:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.399+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c105180 con 0x7f197c0fec70 2024-04-16T07:12:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.399+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f197001ca90 con 0x7f197c0fec70 2024-04-16T07:12:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.399+0000 7f1976000700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 0x7f1964038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.399+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1970049340 con 0x7f197c0fec70 2024-04-16T07:12:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.400+0000 7f1981400700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 0x7f1964038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:36.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.400+0000 7f1981400700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 0x7f1964038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f196c009990 tx=0x7f196c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:36.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.405+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1970011960 con 0x7f197c0fec70 2024-04-16T07:12:36.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.633+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f197c110790 con 0x7f19640368d0 2024-04-16T07:12:36.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.635+0000 7f1976000700 1 -- 172.21.15.26:0/1400838980 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f197c110790 con 0x7f19640368d0 2024-04-16T07:12:36.634 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:12:36.634 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-04-16T07:12:36.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 msgr2=0x7f1964038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:36.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 0x7f1964038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f196c009990 tx=0x7f196c006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 msgr2=0x7f197c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1970009680 tx=0x7f1970009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 shutdown_connections 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f19640368d0 0x7f1964038d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 --2- 172.21.15.26:0/1400838980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f197c0fec70 0x7f197c10f2f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 >> 172.21.15.26:0/1400838980 conn(0x7f197c0fa5d0 msgr2=0x7f197c0fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.638+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 shutdown_connections 2024-04-16T07:12:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:36.639+0000 7f1984800700 1 -- 172.21.15.26:0/1400838980 wait complete. 2024-04-16T07:12:37.962 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:37 smithi026 ceph-mon[28805]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:37.962 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:37 smithi026 ceph-mon[28805]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T07:12:38.025 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-16T07:12:38.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-16T07:12:38.781 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:12:39.954 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:39 smithi026 ceph-mon[28805]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 -- 172.21.15.26:0/2650109902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 msgr2=0x7f28b00ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 --2- 172.21.15.26:0/2650109902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b00ff080 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f28a00048c0 tx=0x7f28a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 -- 172.21.15.26:0/2650109902 shutdown_connections 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 --2- 172.21.15.26:0/2650109902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b00ff080 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 -- 172.21.15.26:0/2650109902 >> 172.21.15.26:0/2650109902 conn(0x7f28b00fa5d0 msgr2=0x7f28b00fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 -- 172.21.15.26:0/2650109902 shutdown_connections 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.489+0000 7f28b5600700 1 -- 172.21.15.26:0/2650109902 wait complete. 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.490+0000 7f28b5600700 1 Processor -- start 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.490+0000 7f28b5600700 1 -- start start 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.491+0000 7f28b5600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.491+0000 7f28b5600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b010f7f0 con 0x7f28b00fec70 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.491+0000 7f28aea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.491+0000 7f28aea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49238/0 (socket says 172.21.15.26:49238) 2024-04-16T07:12:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.491+0000 7f28aea00700 1 -- 172.21.15.26:0/88689759 learned_addr learned my addr 172.21.15.26:0/88689759 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28aea00700 1 -- 172.21.15.26:0/88689759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a0005040 con 0x7f28b00fec70 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28aea00700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009680 tx=0x7f28a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a000b5c0 con 0x7f28b00fec70 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28a000be70 con 0x7f28b00fec70 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b010f9f0 con 0x7f28b00fec70 2024-04-16T07:12:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a00135a0 con 0x7f28b00fec70 2024-04-16T07:12:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.492+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b0110560 con 0x7f28b00fec70 2024-04-16T07:12:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.493+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b0105180 con 0x7f28b00fec70 2024-04-16T07:12:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.493+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f28a001ca90 con 0x7f28b00fec70 2024-04-16T07:12:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.494+0000 7f28a6a00700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 0x7f2894038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.494+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f28a0049410 con 0x7f28b00fec70 2024-04-16T07:12:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.494+0000 7f28ae000700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 0x7f2894038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.495+0000 7f28ae000700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 0x7f2894038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f289c009990 tx=0x7f289c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:41.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.499+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f28a0011940 con 0x7f28b00fec70 2024-04-16T07:12:41.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:41.741+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f28b0111030 con 0x7f28b00fec70 2024-04-16T07:12:42.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:41 smithi026 ceph-mon[28805]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:42.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.675+0000 7f28a6a00700 1 -- 172.21.15.26:0/88689759 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f28a0021090 con 0x7f28b00fec70 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 msgr2=0x7f2894038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 0x7f2894038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f289c009990 tx=0x7f289c006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 msgr2=0x7f28b010f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009680 tx=0x7f28a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 shutdown_connections 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f28940368d0 0x7f2894038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 --2- 172.21.15.26:0/88689759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28b00fec70 0x7f28b010f2b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.679+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 >> 172.21.15.26:0/88689759 conn(0x7f28b00fa5d0 msgr2=0x7f28b00fb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:42.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.680+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 shutdown_connections 2024-04-16T07:12:42.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:12:42.680+0000 7f28b5600700 1 -- 172.21.15.26:0/88689759 wait complete. 2024-04-16T07:12:42.680 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-04-16T07:12:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/88689759' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-16T07:12:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T07:12:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:42 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:43.417 INFO:tasks.cephadm:Adding mon.smithi026 on smithi026 2024-04-16T07:12:43.417 INFO:tasks.cephadm:Adding mon.smithi104 on smithi104 2024-04-16T07:12:43.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi026:172.21.15.26=smithi026;smithi104:172.21.15.104=smithi104' 2024-04-16T07:12:43.862 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:43.974 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:43 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T07:12:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:43 smithi026 ceph-mon[28805]: Deploying daemon grafana.smithi026 on smithi026 2024-04-16T07:12:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:43 smithi026 ceph-mon[28805]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:43 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/88689759' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-16T07:12:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:43 smithi026 ceph-mon[28805]: osdmap e4: 0 total, 0 up, 0 in 2024-04-16T07:12:46.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:45 smithi026 ceph-mon[28805]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:45 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 -- 172.21.15.104:0/383637693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 msgr2=0x7fbff8072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 --2- 172.21.15.104:0/383637693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8072c30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80048c0 tx=0x7fbfe8009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 -- 172.21.15.104:0/383637693 shutdown_connections 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 --2- 172.21.15.104:0/383637693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8072c30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 -- 172.21.15.104:0/383637693 >> 172.21.15.104:0/383637693 conn(0x7fbff80fa5d0 msgr2=0x7fbff80fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 -- 172.21.15.104:0/383637693 shutdown_connections 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.104+0000 7fbffe400700 1 -- 172.21.15.104:0/383637693 wait complete. 2024-04-16T07:12:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.105+0000 7fbffe400700 1 Processor -- start 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.105+0000 7fbffe400700 1 -- start start 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.105+0000 7fbffe400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.105+0000 7fbffe400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff8113c40 con 0x7fbff8072820 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.106+0000 7fbff7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.106+0000 7fbff7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:50976/0 (socket says 172.21.15.104:50976) 2024-04-16T07:12:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.106+0000 7fbff7400700 1 -- 172.21.15.104:0/1771623209 learned_addr learned my addr 172.21.15.104:0/1771623209 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:12:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.107+0000 7fbff7400700 1 -- 172.21.15.104:0/1771623209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8005040 con 0x7fbff8072820 2024-04-16T07:12:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbff7400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009680 tx=0x7fbfe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfe800bed0 con 0x7fbff8072820 2024-04-16T07:12:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfe800b5c0 con 0x7fbff8072820 2024-04-16T07:12:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff8113e40 con 0x7fbff8072820 2024-04-16T07:12:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfe801c5b0 con 0x7fbff8072820 2024-04-16T07:12:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.108+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff81149b0 con 0x7fbff8072820 2024-04-16T07:12:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.109+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff8109570 con 0x7fbff8072820 2024-04-16T07:12:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.110+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fbfe8004d30 con 0x7fbff8072820 2024-04-16T07:12:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.111+0000 7fbfef400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 0x7fbfdc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.111+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbfe804a490 con 0x7fbff8072820 2024-04-16T07:12:46.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.112+0000 7fbff6a00700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 0x7fbfdc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:46.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.113+0000 7fbff6a00700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 0x7fbfdc038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009990 tx=0x7fbfe4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.126+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbfe801b390 con 0x7fbff8072820 2024-04-16T07:12:46.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.347+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 --> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi026:172.21.15.26=smithi026;smithi104:172.21.15.104=smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7fbff8114cb0 con 0x7fbfdc036880 2024-04-16T07:12:46.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.366+0000 7fbfef400700 1 -- 172.21.15.104:0/1771623209 <== mgr.14158 v2:172.21.15.26:6800/2140801142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbff8114cb0 con 0x7fbfdc036880 2024-04-16T07:12:46.365 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 msgr2=0x7fbfdc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 0x7fbfdc038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009990 tx=0x7fbfe4006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 msgr2=0x7fbff8113700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009680 tx=0x7fbfe8009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 shutdown_connections 2024-04-16T07:12:46.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fbfdc036880 0x7fbfdc038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:46.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 --2- 172.21.15.104:0/1771623209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbff8072820 0x7fbff8113700 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:46.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 >> 172.21.15.104:0/1771623209 conn(0x7fbff80fa5d0 msgr2=0x7fbff80fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:46.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.368+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 shutdown_connections 2024-04-16T07:12:46.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:46.369+0000 7fbffe400700 1 -- 172.21.15.104:0/1771623209 wait complete. 2024-04-16T07:12:47.092 DEBUG:teuthology.orchestra.run.smithi104:mon.smithi104> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi104.service 2024-04-16T07:12:47.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:12:47.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:12:47.565 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:47.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:47 smithi026 ceph-mon[28805]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi026:172.21.15.26=smithi026;smithi104:172.21.15.104=smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:12:47.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:47 smithi026 ceph-mon[28805]: Saving service mon spec with placement smithi026:172.21.15.26=smithi026;smithi104:172.21.15.104=smithi104;count:2 2024-04-16T07:12:47.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:47 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:12:47.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:47 smithi026 ceph-mon[28805]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:49.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 -- 172.21.15.104:0/1272175297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 msgr2=0x7f3dfc0ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1272175297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc0ff080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0048c0 tx=0x7f3dec009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 -- 172.21.15.104:0/1272175297 shutdown_connections 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1272175297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc0ff080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 -- 172.21.15.104:0/1272175297 >> 172.21.15.104:0/1272175297 conn(0x7f3dfc0fa5d0 msgr2=0x7f3dfc0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 -- 172.21.15.104:0/1272175297 shutdown_connections 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.501+0000 7f3e02c00700 1 -- 172.21.15.104:0/1272175297 wait complete. 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.502+0000 7f3e02c00700 1 Processor -- start 2024-04-16T07:12:49.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.502+0000 7f3e02c00700 1 -- start start 2024-04-16T07:12:49.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.502+0000 7f3e02c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:49.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.502+0000 7f3e02c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc10f830 con 0x7f3dfc0fec70 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.503+0000 7f3dfbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.503+0000 7f3dfbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:52466/0 (socket says 172.21.15.104:52466) 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.503+0000 7f3dfbe00700 1 -- 172.21.15.104:0/1384300514 learned_addr learned my addr 172.21.15.104:0/1384300514 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.503+0000 7f3dfbe00700 1 -- 172.21.15.104:0/1384300514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec005040 con 0x7f3dfc0fec70 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.503+0000 7f3dfbe00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0096b0 tx=0x7f3dec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.504+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dec00bed0 con 0x7f3dfc0fec70 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.504+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dec00b5c0 con 0x7f3dfc0fec70 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.504+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc10fa30 con 0x7f3dfc0fec70 2024-04-16T07:12:49.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.504+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dec01c5b0 con 0x7f3dfc0fec70 2024-04-16T07:12:49.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.504+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc110550 con 0x7f3dfc0fec70 2024-04-16T07:12:49.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.505+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc105180 con 0x7f3dfc0fec70 2024-04-16T07:12:49.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.506+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f3dec013a80 con 0x7f3dfc0fec70 2024-04-16T07:12:49.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.506+0000 7f3df3e00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 0x7f3de0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:49.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.506+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3dec049ac0 con 0x7f3dfc0fec70 2024-04-16T07:12:49.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.507+0000 7f3dfb400700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 0x7f3de0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:49.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.508+0000 7f3dfb400700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 0x7f3de0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009990 tx=0x7f3de8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:49.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.522+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3dec011960 con 0x7f3dfc0fec70 2024-04-16T07:12:49.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.811+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3dfc110f60 con 0x7f3dfc0fec70 2024-04-16T07:12:49.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.812+0000 7f3df3e00700 1 -- 172.21.15.104:0/1384300514 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3dec011370 con 0x7f3dfc0fec70 2024-04-16T07:12:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:12:49.812 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 msgr2=0x7f3de0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 0x7f3de0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009990 tx=0x7f3de8006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 msgr2=0x7f3dfc10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0096b0 tx=0x7f3dec009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 shutdown_connections 2024-04-16T07:12:49.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f3de0036880 0x7f3de0038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 --2- 172.21.15.104:0/1384300514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dfc0fec70 0x7f3dfc10f2f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 >> 172.21.15.104:0/1384300514 conn(0x7f3dfc0fa5d0 msgr2=0x7f3dfc0fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 shutdown_connections 2024-04-16T07:12:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:49.815+0000 7f3e02c00700 1 -- 172.21.15.104:0/1384300514 wait complete. 2024-04-16T07:12:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:12:50.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:49 smithi026 ceph-mon[28805]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:50.976 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:50 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/1384300514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:12:51.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:12:51.474 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:12:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:51 smithi026 ceph-mon[28805]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:52.022 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:53.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 -- 172.21.15.104:0/577644557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 msgr2=0x7f8c340ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:53.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 --2- 172.21.15.104:0/577644557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c340ffa20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c280048c0 tx=0x7f8c28009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:53.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 -- 172.21.15.104:0/577644557 shutdown_connections 2024-04-16T07:12:53.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 --2- 172.21.15.104:0/577644557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c340ffa20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:53.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 -- 172.21.15.104:0/577644557 >> 172.21.15.104:0/577644557 conn(0x7f8c340f70c0 msgr2=0x7f8c340f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:53.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 -- 172.21.15.104:0/577644557 shutdown_connections 2024-04-16T07:12:53.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.916+0000 7f8c3c800700 1 -- 172.21.15.104:0/577644557 wait complete. 2024-04-16T07:12:53.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.918+0000 7f8c3c800700 1 Processor -- start 2024-04-16T07:12:53.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.918+0000 7f8c3c800700 1 -- start start 2024-04-16T07:12:53.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.918+0000 7f8c3c800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:53.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.918+0000 7f8c3c800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3410f840 con 0x7f8c340fd640 2024-04-16T07:12:53.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.919+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.919+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:52478/0 (socket says 172.21.15.104:52478) 2024-04-16T07:12:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.919+0000 7f8c39e00700 1 -- 172.21.15.104:0/59523367 learned_addr learned my addr 172.21.15.104:0/59523367 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:12:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.919+0000 7f8c39e00700 1 -- 172.21.15.104:0/59523367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c28005040 con 0x7f8c340fd640 2024-04-16T07:12:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.920+0000 7f8c39e00700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800b900 tx=0x7f8c28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.921+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c2800b5c0 con 0x7f8c340fd640 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.921+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3410fa40 con 0x7f8c340fd640 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.921+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c2800be70 con 0x7f8c340fd640 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.921+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c2801c590 con 0x7f8c340fd640 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.921+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c341105b0 con 0x7f8c340fd640 2024-04-16T07:12:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.922+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c34102f50 con 0x7f8c340fd640 2024-04-16T07:12:53.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.923+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f8c28004d30 con 0x7f8c340fd640 2024-04-16T07:12:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.923+0000 7f8c2e000700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 0x7f8c1c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.923+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8c28049820 con 0x7f8c340fd640 2024-04-16T07:12:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.924+0000 7f8c39400700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 0x7f8c1c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:53.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.925+0000 7f8c39400700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 0x7f8c1c038d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8c24009990 tx=0x7f8c24006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:53.932+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8c28011370 con 0x7f8c340fd640 2024-04-16T07:12:54.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:53 smithi026 ceph-mon[28805]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:54.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.226+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c341107f0 con 0x7f8c340fd640 2024-04-16T07:12:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.228+0000 7f8c2e000700 1 -- 172.21.15.104:0/59523367 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8c28021030 con 0x7f8c340fd640 2024-04-16T07:12:54.227 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:12:54.227 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:12:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 msgr2=0x7f8c1c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 0x7f8c1c038d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8c24009990 tx=0x7f8c24006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 msgr2=0x7f8c3410f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800b900 tx=0x7f8c28009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 shutdown_connections 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c1c0368d0 0x7f8c1c038d80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 --2- 172.21.15.104:0/59523367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c340fd640 0x7f8c3410f300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 >> 172.21.15.104:0/59523367 conn(0x7f8c340f70c0 msgr2=0x7f8c340f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.230+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 shutdown_connections 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:54.231+0000 7f8c3c800700 1 -- 172.21.15.104:0/59523367 wait complete. 2024-04-16T07:12:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:12:55.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:54 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/59523367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:12:55.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:12:55.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:12:56.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:55 smithi026 ceph-mon[28805]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:56.356 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:56.454 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:12:58.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:57 smithi026 ceph-mon[28805]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:12:58.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 -- 172.21.15.104:0/569251161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 msgr2=0x7fb7480ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 --2- 172.21.15.104:0/569251161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb7480ff080 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb7380048c0 tx=0x7fb738009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 -- 172.21.15.104:0/569251161 shutdown_connections 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 --2- 172.21.15.104:0/569251161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb7480ff080 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 -- 172.21.15.104:0/569251161 >> 172.21.15.104:0/569251161 conn(0x7fb7480fa5d0 msgr2=0x7fb7480fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 -- 172.21.15.104:0/569251161 shutdown_connections 2024-04-16T07:12:58.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.214+0000 7fb74d600700 1 -- 172.21.15.104:0/569251161 wait complete. 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.215+0000 7fb74d600700 1 Processor -- start 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.215+0000 7fb74d600700 1 -- start start 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.215+0000 7fb74d600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.215+0000 7fb74d600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810f830 con 0x7fb7480fec70 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.216+0000 7fb746a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:58.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.216+0000 7fb746a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:52500/0 (socket says 172.21.15.104:52500) 2024-04-16T07:12:58.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.216+0000 7fb746a00700 1 -- 172.21.15.104:0/872073986 learned_addr learned my addr 172.21.15.104:0/872073986 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:12:58.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.216+0000 7fb746a00700 1 -- 172.21.15.104:0/872073986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738005040 con 0x7fb7480fec70 2024-04-16T07:12:58.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.217+0000 7fb746a00700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb7380096b0 tx=0x7fb738009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:58.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.217+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73800bed0 con 0x7fb7480fec70 2024-04-16T07:12:58.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.217+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb73800b5c0 con 0x7fb7480fec70 2024-04-16T07:12:58.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.217+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74810fa30 con 0x7fb7480fec70 2024-04-16T07:12:58.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.217+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73801c5b0 con 0x7fb7480fec70 2024-04-16T07:12:58.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.218+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb748110550 con 0x7fb7480fec70 2024-04-16T07:12:58.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.218+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb748105180 con 0x7fb7480fec70 2024-04-16T07:12:58.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.219+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb738004d30 con 0x7fb7480fec70 2024-04-16T07:12:58.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.220+0000 7fb73ea00700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 0x7fb72c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:12:58.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.220+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb738049aa0 con 0x7fb7480fec70 2024-04-16T07:12:58.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.220+0000 7fb746000700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 0x7fb72c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:12:58.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.222+0000 7fb746000700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 0x7fb72c038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:12:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.238+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb738011960 con 0x7fb7480fec70 2024-04-16T07:12:58.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.539+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb748111020 con 0x7fb7480fec70 2024-04-16T07:12:58.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.541+0000 7fb73ea00700 1 -- 172.21.15.104:0/872073986 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb738011370 con 0x7fb7480fec70 2024-04-16T07:12:58.541 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:12:58.541 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.544+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 msgr2=0x7fb72c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.544+0000 7fb74d600700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 0x7fb72c038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734006e40 comp rx=0 tx=0).stop 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.544+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 msgr2=0x7fb74810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.544+0000 7fb74d600700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb7380096b0 tx=0x7fb738009e30 comp rx=0 tx=0).stop 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.544+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 shutdown_connections 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.545+0000 7fb74d600700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fb72c0368d0 0x7fb72c038d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.545+0000 7fb74d600700 1 --2- 172.21.15.104:0/872073986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7480fec70 0x7fb74810f2f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.545+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 >> 172.21.15.104:0/872073986 conn(0x7fb7480fa5d0 msgr2=0x7fb7480fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.545+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 shutdown_connections 2024-04-16T07:12:58.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:12:58.545+0000 7fb74d600700 1 -- 172.21.15.104:0/872073986 wait complete. 2024-04-16T07:12:58.545 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:12:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/872073986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:12:59.992 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:12:59 smithi026 ceph-mon[28805]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:00.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:00.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:00.625 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:01.945 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:01 smithi026 ceph-mon[28805]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.529+0000 7ff6ff200700 1 -- 172.21.15.104:0/1451886866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 msgr2=0x7ff6f8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.529+0000 7ff6ff200700 1 --2- 172.21.15.104:0/1451886866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f8070350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80048c0 tx=0x7ff6e8009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.529+0000 7ff6ff200700 1 -- 172.21.15.104:0/1451886866 shutdown_connections 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.529+0000 7ff6ff200700 1 --2- 172.21.15.104:0/1451886866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f8070350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.529+0000 7ff6ff200700 1 -- 172.21.15.104:0/1451886866 >> 172.21.15.104:0/1451886866 conn(0x7ff6f80f81c0 msgr2=0x7ff6f80fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.530+0000 7ff6ff200700 1 -- 172.21.15.104:0/1451886866 shutdown_connections 2024-04-16T07:13:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.530+0000 7ff6ff200700 1 -- 172.21.15.104:0/1451886866 wait complete. 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.531+0000 7ff6ff200700 1 Processor -- start 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.531+0000 7ff6ff200700 1 -- start start 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.531+0000 7ff6ff200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.531+0000 7ff6ff200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f810f830 con 0x7ff6f8071e40 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.532+0000 7ff6f7e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.532+0000 7ff6f7e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:54768/0 (socket says 172.21.15.104:54768) 2024-04-16T07:13:02.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.532+0000 7ff6f7e00700 1 -- 172.21.15.104:0/3475946706 learned_addr learned my addr 172.21.15.104:0/3475946706 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:02.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.533+0000 7ff6f7e00700 1 -- 172.21.15.104:0/3475946706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8005040 con 0x7ff6f8071e40 2024-04-16T07:13:02.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.533+0000 7ff6f7e00700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800b900 tx=0x7ff6e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.533+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6e800b5c0 con 0x7ff6f8071e40 2024-04-16T07:13:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.534+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6e800be70 con 0x7ff6f8071e40 2024-04-16T07:13:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.534+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f810fa30 con 0x7ff6f8071e40 2024-04-16T07:13:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.534+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6e801c590 con 0x7ff6f8071e40 2024-04-16T07:13:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.534+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f81105a0 con 0x7ff6f8071e40 2024-04-16T07:13:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.535+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8105180 con 0x7ff6f8071e40 2024-04-16T07:13:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.535+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7ff6e8004d30 con 0x7ff6f8071e40 2024-04-16T07:13:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.536+0000 7ff6efe00700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 0x7ff6dc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.536+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff6e8049440 con 0x7ff6f8071e40 2024-04-16T07:13:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.536+0000 7ff6f7400700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 0x7ff6dc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.538+0000 7ff6f7400700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 0x7ff6dc038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009990 tx=0x7ff6e4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:02.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.552+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff6e801b350 con 0x7ff6f8071e40 2024-04-16T07:13:02.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.843+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6f805f5f0 con 0x7ff6f8071e40 2024-04-16T07:13:02.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.844+0000 7ff6efe00700 1 -- 172.21.15.104:0/3475946706 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff6e8021020 con 0x7ff6f8071e40 2024-04-16T07:13:02.843 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:02.843 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 msgr2=0x7ff6dc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 0x7ff6dc038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009990 tx=0x7ff6e4006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 msgr2=0x7ff6f810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800b900 tx=0x7ff6e8009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 shutdown_connections 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7ff6dc0368d0 0x7ff6dc038d80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 --2- 172.21.15.104:0/3475946706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8071e40 0x7ff6f810f2f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:02.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 >> 172.21.15.104:0/3475946706 conn(0x7ff6f80f81c0 msgr2=0x7ff6f8105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:02.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 shutdown_connections 2024-04-16T07:13:02.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:02.847+0000 7ff6ff200700 1 -- 172.21.15.104:0/3475946706 wait complete. 2024-04-16T07:13:02.847 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:03.858 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:03 smithi026 ceph-mon[28805]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:03.859 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:03 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3475946706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:04.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:04.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:05.012 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:05.131 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:04 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:05 smithi026 ceph-mon[28805]: Deploying daemon prometheus.smithi026 on smithi026 2024-04-16T07:13:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:05 smithi026 ceph-mon[28805]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:05 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:07.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 -- 172.21.15.104:0/3545454260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 msgr2=0x7f70a00fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:07.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 --2- 172.21.15.104:0/3545454260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a00fce80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f70900048c0 tx=0x7f7090009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:07.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 -- 172.21.15.104:0/3545454260 shutdown_connections 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 --2- 172.21.15.104:0/3545454260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a00fce80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 -- 172.21.15.104:0/3545454260 >> 172.21.15.104:0/3545454260 conn(0x7f70a00f81d0 msgr2=0x7f70a00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 -- 172.21.15.104:0/3545454260 shutdown_connections 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.022+0000 7f70a4e00700 1 -- 172.21.15.104:0/3545454260 wait complete. 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.024+0000 7f70a4e00700 1 Processor -- start 2024-04-16T07:13:07.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.024+0000 7f70a4e00700 1 -- start start 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.024+0000 7f70a4e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.024+0000 7f70a4e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a010d600 con 0x7f70a00fca70 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.025+0000 7f709de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.025+0000 7f709de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:54782/0 (socket says 172.21.15.104:54782) 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.025+0000 7f709de00700 1 -- 172.21.15.104:0/871604051 learned_addr learned my addr 172.21.15.104:0/871604051 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.025+0000 7f709de00700 1 -- 172.21.15.104:0/871604051 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7090005040 con 0x7f70a00fca70 2024-04-16T07:13:07.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.026+0000 7f709de00700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70900096b0 tx=0x7f7090009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:07.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.026+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f709000bed0 con 0x7f70a00fca70 2024-04-16T07:13:07.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.026+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f709000b5c0 con 0x7f70a00fca70 2024-04-16T07:13:07.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.026+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a010d800 con 0x7f70a00fca70 2024-04-16T07:13:07.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.027+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f709001c5b0 con 0x7f70a00fca70 2024-04-16T07:13:07.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.027+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a010e370 con 0x7f70a00fca70 2024-04-16T07:13:07.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.028+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0102f80 con 0x7f70a00fca70 2024-04-16T07:13:07.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.029+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f7090004d30 con 0x7f70a00fca70 2024-04-16T07:13:07.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.029+0000 7f7096000700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 0x7f7084038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.029+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f709004a5e0 con 0x7f70a00fca70 2024-04-16T07:13:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.030+0000 7f709d400700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 0x7f7084038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.031+0000 7f709d400700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 0x7f7084038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f708c009990 tx=0x7f708c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:07.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.043+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f709001b980 con 0x7f70a00fca70 2024-04-16T07:13:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.336+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70a004bed0 con 0x7f70a00fca70 2024-04-16T07:13:07.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.337+0000 7f7096000700 1 -- 172.21.15.104:0/871604051 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f709001b390 con 0x7f70a00fca70 2024-04-16T07:13:07.336 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:07.336 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 msgr2=0x7f7084038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 0x7f7084038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f708c009990 tx=0x7f708c006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 msgr2=0x7f70a010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70900096b0 tx=0x7f7090009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 shutdown_connections 2024-04-16T07:13:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f7084036880 0x7f7084038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 --2- 172.21.15.104:0/871604051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a00fca70 0x7f70a010d0c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 >> 172.21.15.104:0/871604051 conn(0x7f70a00f81d0 msgr2=0x7f70a0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 shutdown_connections 2024-04-16T07:13:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:07.340+0000 7f70a4e00700 1 -- 172.21.15.104:0/871604051 wait complete. 2024-04-16T07:13:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:08.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:07 smithi026 ceph-mon[28805]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:08.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:07 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/871604051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:09.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:09.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:09.519 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:09.623 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:10.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:09 smithi026 ceph-mon[28805]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:11.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 -- 172.21.15.104:0/3153923819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 msgr2=0x7f2b580ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 --2- 172.21.15.104:0/3153923819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580ff480 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b480048c0 tx=0x7f2b48009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 -- 172.21.15.104:0/3153923819 shutdown_connections 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 --2- 172.21.15.104:0/3153923819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580ff480 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 -- 172.21.15.104:0/3153923819 >> 172.21.15.104:0/3153923819 conn(0x7f2b580f6ef0 msgr2=0x7f2b580f9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.458+0000 7f2b57e00700 1 -- 172.21.15.104:0/3153923819 shutdown_connections 2024-04-16T07:13:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.459+0000 7f2b57e00700 1 -- 172.21.15.104:0/3153923819 wait complete. 2024-04-16T07:13:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.460+0000 7f2b57e00700 1 Processor -- start 2024-04-16T07:13:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.460+0000 7f2b57e00700 1 -- start start 2024-04-16T07:13:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.460+0000 7f2b57e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.460+0000 7f2b57e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580fd310 con 0x7f2b58065d60 2024-04-16T07:13:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.461+0000 7f2b56a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.461+0000 7f2b56a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:55714/0 (socket says 172.21.15.104:55714) 2024-04-16T07:13:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.461+0000 7f2b56a00700 1 -- 172.21.15.104:0/4081191153 learned_addr learned my addr 172.21.15.104:0/4081191153 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.461+0000 7f2b56a00700 1 -- 172.21.15.104:0/4081191153 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48005040 con 0x7f2b58065d60 2024-04-16T07:13:11.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b56a00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800b900 tx=0x7f2b48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4800b5c0 con 0x7f2b58065d60 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b4800be70 con 0x7f2b58065d60 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4801c590 con 0x7f2b58065d60 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b580fb3e0 con 0x7f2b58065d60 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.462+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b580fb8d0 con 0x7f2b58065d60 2024-04-16T07:13:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.463+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5805f5f0 con 0x7f2b58065d60 2024-04-16T07:13:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.464+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f2b48004d30 con 0x7f2b58065d60 2024-04-16T07:13:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.464+0000 7f2b4ea00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 0x7f2b3c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.464+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2b48049440 con 0x7f2b58065d60 2024-04-16T07:13:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.465+0000 7f2b56000700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 0x7f2b3c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.466+0000 7f2b56000700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 0x7f2b3c038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009990 tx=0x7f2b44006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:11.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.480+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2b4801b350 con 0x7f2b58065d60 2024-04-16T07:13:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.776+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b58106260 con 0x7f2b58065d60 2024-04-16T07:13:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.777+0000 7f2b4ea00700 1 -- 172.21.15.104:0/4081191153 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2b48021020 con 0x7f2b58065d60 2024-04-16T07:13:11.776 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:11.776 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.779+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 msgr2=0x7f2b3c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 0x7f2b3c038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009990 tx=0x7f2b44006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 msgr2=0x7f2b580fcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800b900 tx=0x7f2b48009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 shutdown_connections 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f2b3c0368d0 0x7f2b3c038d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 --2- 172.21.15.104:0/4081191153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b58065d60 0x7f2b580fcdd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 >> 172.21.15.104:0/4081191153 conn(0x7f2b580f6ef0 msgr2=0x7f2b581039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 shutdown_connections 2024-04-16T07:13:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:11.780+0000 7f2b57e00700 1 -- 172.21.15.104:0/4081191153 wait complete. 2024-04-16T07:13:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:12.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:11 smithi026 ceph-mon[28805]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:12.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:11 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/4081191153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:13.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:12 smithi026 ceph-mon[28805]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:13.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:13.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:14.008 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:16.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:15 smithi026 ceph-mon[28805]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:16.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 -- 172.21.15.104:0/3792214909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 msgr2=0x7f26e0103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:16.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3792214909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e0103490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f26d00048c0 tx=0x7f26d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:16.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 -- 172.21.15.104:0/3792214909 shutdown_connections 2024-04-16T07:13:16.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3792214909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e0103490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:16.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 -- 172.21.15.104:0/3792214909 >> 172.21.15.104:0/3792214909 conn(0x7f26e0072300 msgr2=0x7f26e0074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:16.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 -- 172.21.15.104:0/3792214909 shutdown_connections 2024-04-16T07:13:16.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.111+0000 7f26e4e00700 1 -- 172.21.15.104:0/3792214909 wait complete. 2024-04-16T07:13:16.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.112+0000 7f26e4e00700 1 Processor -- start 2024-04-16T07:13:16.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.113+0000 7f26e4e00700 1 -- start start 2024-04-16T07:13:16.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.113+0000 7f26e4e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:16.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.113+0000 7f26e4e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0113c00 con 0x7f26e0103080 2024-04-16T07:13:16.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.114+0000 7f26dde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:16.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.114+0000 7f26dde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:55724/0 (socket says 172.21.15.104:55724) 2024-04-16T07:13:16.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.114+0000 7f26dde00700 1 -- 172.21.15.104:0/3418150557 learned_addr learned my addr 172.21.15.104:0/3418150557 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:16.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.114+0000 7f26dde00700 1 -- 172.21.15.104:0/3418150557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0005040 con 0x7f26e0103080 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26dde00700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009680 tx=0x7f26d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d000b5c0 con 0x7f26e0103080 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0113e00 con 0x7f26e0103080 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d000be70 con 0x7f26e0103080 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d00135a0 con 0x7f26e0103080 2024-04-16T07:13:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.115+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e0114970 con 0x7f26e0103080 2024-04-16T07:13:16.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.116+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e0109570 con 0x7f26e0103080 2024-04-16T07:13:16.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.117+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f26d0004d30 con 0x7f26e0103080 2024-04-16T07:13:16.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.117+0000 7f26d6000700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 0x7f26c4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:16.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.117+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f26d0049480 con 0x7f26e0103080 2024-04-16T07:13:16.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.118+0000 7f26dd400700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 0x7f26c4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:16.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.119+0000 7f26dd400700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 0x7f26c4038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009990 tx=0x7f26cc006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:16.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.132+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f26d0011780 con 0x7f26e0103080 2024-04-16T07:13:16.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.405+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26e0115440 con 0x7f26e0103080 2024-04-16T07:13:16.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.406+0000 7f26d6000700 1 -- 172.21.15.104:0/3418150557 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f26d0021300 con 0x7f26e0103080 2024-04-16T07:13:16.405 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:16.406 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 msgr2=0x7f26c4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 0x7f26c4038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009990 tx=0x7f26cc006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 msgr2=0x7f26e01136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009680 tx=0x7f26d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 shutdown_connections 2024-04-16T07:13:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f26c4036880 0x7f26c4038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:16.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 --2- 172.21.15.104:0/3418150557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e0103080 0x7f26e01136c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:16.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 >> 172.21.15.104:0/3418150557 conn(0x7f26e0072300 msgr2=0x7f26e0074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:16.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.409+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 shutdown_connections 2024-04-16T07:13:16.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:16.410+0000 7f26e4e00700 1 -- 172.21.15.104:0/3418150557 wait complete. 2024-04-16T07:13:16.409 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:17.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:16 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3418150557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:17.913 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:17 smithi026 ceph-mon[28805]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:18.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:18.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:18.631 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:18.729 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:20.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:19 smithi026 ceph-mon[28805]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.729+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1422084150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 msgr2=0x7f4ff00ff6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.729+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1422084150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00ff6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00048c0 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.730+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1422084150 shutdown_connections 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.730+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1422084150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00ff6f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.730+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1422084150 >> 172.21.15.104:0/1422084150 conn(0x7f4ff00f70c0 msgr2=0x7f4ff00f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.730+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1422084150 shutdown_connections 2024-04-16T07:13:20.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.730+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1422084150 wait complete. 2024-04-16T07:13:20.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.731+0000 7f4ff6c00700 1 Processor -- start 2024-04-16T07:13:20.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.731+0000 7f4ff6c00700 1 -- start start 2024-04-16T07:13:20.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.731+0000 7f4ff6c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:20.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.731+0000 7f4ff6c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00fd580 con 0x7f4ff0066930 2024-04-16T07:13:20.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.732+0000 7f4fefe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:20.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.732+0000 7f4fefe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:44436/0 (socket says 172.21.15.104:44436) 2024-04-16T07:13:20.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.732+0000 7f4fefe00700 1 -- 172.21.15.104:0/1799568361 learned_addr learned my addr 172.21.15.104:0/1799568361 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:20.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.732+0000 7f4fefe00700 1 -- 172.21.15.104:0/1799568361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0005040 con 0x7f4ff0066930 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4fefe00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009480 tx=0x7f4fe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe000bba0 con 0x7f4ff0066930 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff00fb650 con 0x7f4ff0066930 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff00fbb40 con 0x7f4ff0066930 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fe000bd00 con 0x7f4ff0066930 2024-04-16T07:13:20.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.733+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe001c550 con 0x7f4ff0066930 2024-04-16T07:13:20.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.734+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff005f5f0 con 0x7f4ff0066930 2024-04-16T07:13:20.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.735+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f4fe0013a80 con 0x7f4ff0066930 2024-04-16T07:13:20.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.735+0000 7f4fe7e00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 0x7f4fd4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:20.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.735+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4fe00495f0 con 0x7f4ff0066930 2024-04-16T07:13:20.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.736+0000 7f4fef400700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 0x7f4fd4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:20.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.737+0000 7f4fef400700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 0x7f4fd4038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009990 tx=0x7f4fdc006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:20.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:20.751+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4fe0013420 con 0x7f4ff0066930 2024-04-16T07:13:21.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.047+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ff0106440 con 0x7f4ff0066930 2024-04-16T07:13:21.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.049+0000 7f4fe7e00700 1 -- 172.21.15.104:0/1799568361 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4fe0021030 con 0x7f4ff0066930 2024-04-16T07:13:21.047 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:21.048 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 msgr2=0x7f4fd4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 0x7f4fd4038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009990 tx=0x7f4fdc006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 msgr2=0x7f4ff00fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009480 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 shutdown_connections 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f4fd4036880 0x7f4fd4038d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 --2- 172.21.15.104:0/1799568361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ff0066930 0x7f4ff00fd040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 >> 172.21.15.104:0/1799568361 conn(0x7f4ff00f70c0 msgr2=0x7f4ff0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 shutdown_connections 2024-04-16T07:13:21.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:21.052+0000 7f4ff6c00700 1 -- 172.21.15.104:0/1799568361 wait complete. 2024-04-16T07:13:21.052 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:22.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:22.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/1799568361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:21 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-16T07:13:22.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:22.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:23.367 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:23.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:23 smithi026 ceph-mon[28805]: from='mgr.14158 172.21.15.26:0/1979212546' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-16T07:13:23.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:23 smithi026 ceph-mon[28805]: mgrmap e14: smithi026.tawexg(active, since 81s) 2024-04-16T07:13:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 -- 172.21.15.104:0/1124910667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 msgr2=0x7fd7500fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 --2- 172.21.15.104:0/1124910667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd7500fce70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7400048c0 tx=0x7fd740009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 -- 172.21.15.104:0/1124910667 shutdown_connections 2024-04-16T07:13:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 --2- 172.21.15.104:0/1124910667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd7500fce70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 -- 172.21.15.104:0/1124910667 >> 172.21.15.104:0/1124910667 conn(0x7fd7500f81c0 msgr2=0x7fd7500fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 -- 172.21.15.104:0/1124910667 shutdown_connections 2024-04-16T07:13:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.291+0000 7fd756a00700 1 -- 172.21.15.104:0/1124910667 wait complete. 2024-04-16T07:13:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.292+0000 7fd756a00700 1 Processor -- start 2024-04-16T07:13:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd756a00700 1 -- start start 2024-04-16T07:13:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd756a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd756a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75010d5f0 con 0x7fd7500fca60 2024-04-16T07:13:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd74fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd74fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:44468/0 (socket says 172.21.15.104:44468) 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.293+0000 7fd74fe00700 1 -- 172.21.15.104:0/140441690 learned_addr learned my addr 172.21.15.104:0/140441690 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.294+0000 7fd74fe00700 1 -- 172.21.15.104:0/140441690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd740005040 con 0x7fd7500fca60 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.294+0000 7fd74fe00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd740009680 tx=0x7fd740009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.294+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74000b5c0 con 0x7fd7500fca60 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.294+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75010d7f0 con 0x7fd7500fca60 2024-04-16T07:13:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.295+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd74000be70 con 0x7fd7500fca60 2024-04-16T07:13:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.295+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7400135a0 con 0x7fd7500fca60 2024-04-16T07:13:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.295+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75010e360 con 0x7fd7500fca60 2024-04-16T07:13:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.296+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd750102f70 con 0x7fd7500fca60 2024-04-16T07:13:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.296+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7fd740004d30 con 0x7fd7500fca60 2024-04-16T07:13:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.296+0000 7fd747e00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 0x7fd734038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.296+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd7400494a0 con 0x7fd7500fca60 2024-04-16T07:13:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.297+0000 7fd74f400700 1 -- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 msgr2=0x7fd734038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2140801142 2024-04-16T07:13:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.297+0000 7fd74f400700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 0x7fd734038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:13:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.311+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd740011960 con 0x7fd7500fca60 2024-04-16T07:13:25.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.498+0000 7fd74f400700 1 -- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 msgr2=0x7fd734038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2140801142 2024-04-16T07:13:25.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.498+0000 7fd74f400700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 0x7fd734038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:13:25.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.611+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd75010ed70 con 0x7fd7500fca60 2024-04-16T07:13:25.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.612+0000 7fd747e00700 1 -- 172.21.15.104:0/140441690 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd740021070 con 0x7fd7500fca60 2024-04-16T07:13:25.611 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:25.611 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 msgr2=0x7fd734038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 0x7fd734038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 msgr2=0x7fd75010d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd740009680 tx=0x7fd740009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 shutdown_connections 2024-04-16T07:13:25.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7fd7340368d0 0x7fd734038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:25.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 --2- 172.21.15.104:0/140441690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7500fca60 0x7fd75010d0b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:25.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 >> 172.21.15.104:0/140441690 conn(0x7fd7500f81c0 msgr2=0x7fd750103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:25.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.615+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 shutdown_connections 2024-04-16T07:13:25.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:25.616+0000 7fd756a00700 1 -- 172.21.15.104:0/140441690 wait complete. 2024-04-16T07:13:25.615 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:26.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:25 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/140441690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:27.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:27.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:27.733 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:27.831 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 -- 172.21.15.104:0/272504753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 msgr2=0x7f8c7c0ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 --2- 172.21.15.104:0/272504753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c0ff080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c700048c0 tx=0x7f8c70009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 -- 172.21.15.104:0/272504753 shutdown_connections 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 --2- 172.21.15.104:0/272504753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c0ff080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 -- 172.21.15.104:0/272504753 >> 172.21.15.104:0/272504753 conn(0x7f8c7c0fa5d0 msgr2=0x7f8c7c0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 -- 172.21.15.104:0/272504753 shutdown_connections 2024-04-16T07:13:29.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.700+0000 7f8c84200700 1 -- 172.21.15.104:0/272504753 wait complete. 2024-04-16T07:13:29.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.701+0000 7f8c84200700 1 Processor -- start 2024-04-16T07:13:29.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.702+0000 7f8c84200700 1 -- start start 2024-04-16T07:13:29.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.702+0000 7f8c84200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:29.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.702+0000 7f8c84200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c10f7d0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.702+0000 7f8c81800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:29.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.703+0000 7f8c81800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:59818/0 (socket says 172.21.15.104:59818) 2024-04-16T07:13:29.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.703+0000 7f8c81800700 1 -- 172.21.15.104:0/336319597 learned_addr learned my addr 172.21.15.104:0/336319597 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:29.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.703+0000 7f8c81800700 1 -- 172.21.15.104:0/336319597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70005040 con 0x7f8c7c0fec70 2024-04-16T07:13:29.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.703+0000 7f8c81800700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c700096b0 tx=0x7f8c70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:29.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.703+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c7000bed0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.704+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c7000b5c0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.704+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c10f9d0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.704+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c7001c5b0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.704+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c110540 con 0x7f8c7c0fec70 2024-04-16T07:13:29.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.705+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c105180 con 0x7f8c7c0fec70 2024-04-16T07:13:29.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.705+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7f8c70013a80 con 0x7f8c7c0fec70 2024-04-16T07:13:29.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.705+0000 7f8c79600700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 0x7f8c64038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:29.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.706+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8c70049af0 con 0x7f8c7c0fec70 2024-04-16T07:13:29.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.706+0000 7f8c80e00700 1 -- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 msgr2=0x7f8c64038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2140801142 2024-04-16T07:13:29.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.706+0000 7f8c80e00700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 0x7f8c64038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:13:29.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.720+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8c70011960 con 0x7f8c7c0fec70 2024-04-16T07:13:29.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.907+0000 7f8c80e00700 1 -- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 msgr2=0x7f8c64038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2140801142 2024-04-16T07:13:29.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:29.907+0000 7f8c80e00700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 0x7f8c64038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:13:30.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.016+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c7c110f50 con 0x7f8c7c0fec70 2024-04-16T07:13:30.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.017+0000 7f8c79600700 1 -- 172.21.15.104:0/336319597 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8c70011370 con 0x7f8c7c0fec70 2024-04-16T07:13:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:30.018 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 msgr2=0x7f8c64038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 0x7f8c64038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 msgr2=0x7f8c7c10f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c700096b0 tx=0x7f8c70009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 shutdown_connections 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:6800/2140801142,v1:172.21.15.26:6801/2140801142] conn(0x7f8c64036920 0x7f8c64038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:30.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 --2- 172.21.15.104:0/336319597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c7c0fec70 0x7f8c7c10f290 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:30.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 >> 172.21.15.104:0/336319597 conn(0x7f8c7c0fa5d0 msgr2=0x7f8c7c0fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:30.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 shutdown_connections 2024-04-16T07:13:30.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:30.020+0000 7f8c84200700 1 -- 172.21.15.104:0/336319597 wait complete. 2024-04-16T07:13:30.020 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:30.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:30 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/336319597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:31.464 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: Active manager daemon smithi026.tawexg restarted 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: Activating manager daemon smithi026.tawexg 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: mgrmap e15: smithi026.tawexg(active, starting, since 0.00679361s) 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:13:31.465 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:13:31.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:31.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:32.167 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:32.653 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:32.653 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:13:32.653 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:13:32.653 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:13:32.653 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:32.654 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:32 smithi026 ceph-mon[28805]: mgrmap e16: smithi026.tawexg(active, since 1.00855s) 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 -- 172.21.15.104:0/838821242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 msgr2=0x7fdfac0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 --2- 172.21.15.104:0/838821242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac0703b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0048c0 tx=0x7fdf9c009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 -- 172.21.15.104:0/838821242 shutdown_connections 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 --2- 172.21.15.104:0/838821242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac0703b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 -- 172.21.15.104:0/838821242 >> 172.21.15.104:0/838821242 conn(0x7fdfac0f81d0 msgr2=0x7fdfac0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 -- 172.21.15.104:0/838821242 shutdown_connections 2024-04-16T07:13:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.967+0000 7fdfb3200700 1 -- 172.21.15.104:0/838821242 wait complete. 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.968+0000 7fdfb3200700 1 Processor -- start 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.968+0000 7fdfb3200700 1 -- start start 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.968+0000 7fdfb3200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.968+0000 7fdfb3200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac10f820 con 0x7fdfac071ea0 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.969+0000 7fdfabe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.969+0000 7fdfabe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:59842/0 (socket says 172.21.15.104:59842) 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.969+0000 7fdfabe00700 1 -- 172.21.15.104:0/2726004397 learned_addr learned my addr 172.21.15.104:0/2726004397 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:33.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.969+0000 7fdfabe00700 1 -- 172.21.15.104:0/2726004397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c005040 con 0x7fdfac071ea0 2024-04-16T07:13:33.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.969+0000 7fdfabe00700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00b900 tx=0x7fdf9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:33.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.970+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf9c00b5c0 con 0x7fdfac071ea0 2024-04-16T07:13:33.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.970+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf9c00be70 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.970+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac10fa20 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.970+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf9c01c590 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.970+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac110590 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.971+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fdf9c01c800 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.971+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac105190 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.971+0000 7fdfa3e00700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 0x7fdf90038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.971+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fdf9c0496a0 con 0x7fdfac071ea0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.972+0000 7fdfab400700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 0x7fdf90038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.972+0000 7fdfab400700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 0x7fdf90038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009990 tx=0x7fdf98006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:33.981+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdf9c01bd20 con 0x7fdfac071ea0 2024-04-16T07:13:34.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.275+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfac05f5f0 con 0x7fdfac071ea0 2024-04-16T07:13:34.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.276+0000 7fdfa3e00700 1 -- 172.21.15.104:0/2726004397 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fdf9c021030 con 0x7fdfac071ea0 2024-04-16T07:13:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:34.275 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 msgr2=0x7fdf90038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 0x7fdf90038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009990 tx=0x7fdf98006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 msgr2=0x7fdfac10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00b900 tx=0x7fdf9c009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 shutdown_connections 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fdf900369c0 0x7fdf90038e70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 --2- 172.21.15.104:0/2726004397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfac071ea0 0x7fdfac10f2e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:34.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 >> 172.21.15.104:0/2726004397 conn(0x7fdfac0f81d0 msgr2=0x7fdfac105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:34.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 shutdown_connections 2024-04-16T07:13:34.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:34.279+0000 7fdfb3200700 1 -- 172.21.15.104:0/2726004397 wait complete. 2024-04-16T07:13:34.279 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:34.572 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:34 smithi026 ceph-mon[28805]: mgrmap e17: smithi026.tawexg(active, since 2s) 2024-04-16T07:13:34.572 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:34 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:34.572 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:34 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/2726004397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:35.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:35.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:35 smithi026 ceph-mon[28805]: [16/Apr/2024:07:13:34] ENGINE Bus STARTING 2024-04-16T07:13:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:35 smithi026 ceph-mon[28805]: [16/Apr/2024:07:13:34] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:13:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:35 smithi026 ceph-mon[28805]: [16/Apr/2024:07:13:34] ENGINE Bus STARTED 2024-04-16T07:13:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:37.230 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T07:13:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:37 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:37 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.329+0000 7f7948400700 1 -- 172.21.15.104:0/3420267028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 msgr2=0x7f79400fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.329+0000 7f7948400700 1 --2- 172.21.15.104:0/3420267028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f79400fce70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7934009c60 tx=0x7f7934004d10 comp rx=0 tx=0).stop 2024-04-16T07:13:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.329+0000 7f7948400700 1 -- 172.21.15.104:0/3420267028 shutdown_connections 2024-04-16T07:13:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.329+0000 7f7948400700 1 --2- 172.21.15.104:0/3420267028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f79400fce70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.330+0000 7f7948400700 1 -- 172.21.15.104:0/3420267028 >> 172.21.15.104:0/3420267028 conn(0x7f79400f8200 msgr2=0x7f79400fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.330+0000 7f7948400700 1 -- 172.21.15.104:0/3420267028 shutdown_connections 2024-04-16T07:13:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.330+0000 7f7948400700 1 -- 172.21.15.104:0/3420267028 wait complete. 2024-04-16T07:13:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.331+0000 7f7948400700 1 Processor -- start 2024-04-16T07:13:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.331+0000 7f7948400700 1 -- start start 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.331+0000 7f7948400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.331+0000 7f7948400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79400722a0 con 0x7f79400fca60 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.332+0000 7f7945a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.332+0000 7f7945a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:50362/0 (socket says 172.21.15.104:50362) 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.332+0000 7f7945a00700 1 -- 172.21.15.104:0/1422852760 learned_addr learned my addr 172.21.15.104:0/1422852760 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.333+0000 7f7945a00700 1 -- 172.21.15.104:0/1422852760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934009940 con 0x7f79400fca60 2024-04-16T07:13:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.333+0000 7f7945a00700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7934009500 tx=0x7f7934004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.334+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7934007c80 con 0x7f79400fca60 2024-04-16T07:13:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.334+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f793401d910 con 0x7f79400fca60 2024-04-16T07:13:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.334+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940070370 con 0x7f79400fca60 2024-04-16T07:13:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.334+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79340129d0 con 0x7f79400fca60 2024-04-16T07:13:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.334+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940070860 con 0x7f79400fca60 2024-04-16T07:13:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.335+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794005f5f0 con 0x7f79400fca60 2024-04-16T07:13:39.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.336+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f7934019040 con 0x7f79400fca60 2024-04-16T07:13:39.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.336+0000 7f793d600700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 0x7f7928038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:39.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.336+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f793404a980 con 0x7f79400fca60 2024-04-16T07:13:39.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.337+0000 7f7945000700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 0x7f7928038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:39.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.338+0000 7f7945000700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 0x7f7928038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f792c009990 tx=0x7f792c006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:39.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.351+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f793401ab80 con 0x7f79400fca60 2024-04-16T07:13:39.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.649+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f794004bed0 con 0x7f79400fca60 2024-04-16T07:13:39.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.650+0000 7f793d600700 1 -- 172.21.15.104:0/1422852760 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7934017090 con 0x7f79400fca60 2024-04-16T07:13:39.649 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:39.649 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 msgr2=0x7f7928038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 0x7f7928038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f792c009990 tx=0x7f792c006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 msgr2=0x7f7940071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7934009500 tx=0x7f7934004d10 comp rx=0 tx=0).stop 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 shutdown_connections 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f79280369c0 0x7f7928038e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 --2- 172.21.15.104:0/1422852760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79400fca60 0x7f7940071d60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 >> 172.21.15.104:0/1422852760 conn(0x7f79400f8200 msgr2=0x7f7940103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 shutdown_connections 2024-04-16T07:13:39.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:39.653+0000 7f7948400700 1 -- 172.21.15.104:0/1422852760 wait complete. 2024-04-16T07:13:39.653 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:39.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:39.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:39.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:13:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/1422852760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:13:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:13:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:13:41.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:41.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:41 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:13:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:41 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:13:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:41 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:41 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:42 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:13:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:42 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 -- 172.21.15.104:0/2397565823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 msgr2=0x7f4774109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 --2- 172.21.15.104:0/2397565823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774109470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f47680048c0 tx=0x7f4768009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 -- 172.21.15.104:0/2397565823 shutdown_connections 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 --2- 172.21.15.104:0/2397565823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774109470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 -- 172.21.15.104:0/2397565823 >> 172.21.15.104:0/2397565823 conn(0x7f47740728d0 msgr2=0x7f4774074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:43.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.910+0000 7f477c000700 1 -- 172.21.15.104:0/2397565823 shutdown_connections 2024-04-16T07:13:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.911+0000 7f477c000700 1 -- 172.21.15.104:0/2397565823 wait complete. 2024-04-16T07:13:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.911+0000 7f477c000700 1 Processor -- start 2024-04-16T07:13:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.911+0000 7f477c000700 1 -- start start 2024-04-16T07:13:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.912+0000 7f477c000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.912+0000 7f477c000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4774119c70 con 0x7f47741090a0 2024-04-16T07:13:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.913+0000 7f4779600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.913+0000 7f4779600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:50380/0 (socket says 172.21.15.104:50380) 2024-04-16T07:13:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.913+0000 7f4779600700 1 -- 172.21.15.104:0/1895106553 learned_addr learned my addr 172.21.15.104:0/1895106553 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.913+0000 7f4779600700 1 -- 172.21.15.104:0/1895106553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4768005040 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f4779600700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f476800b820 tx=0x7f4768009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47680093c0 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4774119e70 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4768013420 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f476801c5b0 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.914+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47741136b0 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.915+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47741047f0 con 0x7f47741090a0 2024-04-16T07:13:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.916+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f4768013a90 con 0x7f47741090a0 2024-04-16T07:13:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.916+0000 7f4771600700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 0x7f475c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.916+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4768049ff0 con 0x7f47741090a0 2024-04-16T07:13:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.917+0000 7f4778c00700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 0x7f475c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.918+0000 7f4778c00700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 0x7f475c038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4760009990 tx=0x7f4760006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:43.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:43.932+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4768011370 con 0x7f47741090a0 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:13:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T07:13:44.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:13:44.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.229+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f477410d3c0 con 0x7f47741090a0 2024-04-16T07:13:44.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.230+0000 7f4771600700 1 -- 172.21.15.104:0/1895106553 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4768024ed0 con 0x7f47741090a0 2024-04-16T07:13:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:44.229 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:44.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 msgr2=0x7f475c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:44.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 0x7f475c038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4760009990 tx=0x7f4760006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:44.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 msgr2=0x7f4774119730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f476800b820 tx=0x7f4768009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 shutdown_connections 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f475c036970 0x7f475c038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 --2- 172.21.15.104:0/1895106553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47741090a0 0x7f4774119730 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 >> 172.21.15.104:0/1895106553 conn(0x7f47740728d0 msgr2=0x7f4774073170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.232+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 shutdown_connections 2024-04-16T07:13:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:44.233+0000 7f477c000700 1 -- 172.21.15.104:0/1895106553 wait complete. 2024-04-16T07:13:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:45.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:44 smithi026 ceph-mon[28805]: Deploying daemon crash.smithi104 on smithi104 2024-04-16T07:13:45.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:44 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/1895106553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:46.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:46.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:46.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:13:47.022 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.286+0000 7f6361e00700 1 -- 172.21.15.104:0/3389523726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 msgr2=0x7f635c0728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.286+0000 7f6361e00700 1 --2- 172.21.15.104:0/3389523726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c0728c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f634c0048c0 tx=0x7f634c009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.286+0000 7f6361e00700 1 -- 172.21.15.104:0/3389523726 shutdown_connections 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.286+0000 7f6361e00700 1 --2- 172.21.15.104:0/3389523726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c0728c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.286+0000 7f6361e00700 1 -- 172.21.15.104:0/3389523726 >> 172.21.15.104:0/3389523726 conn(0x7f635c0fa950 msgr2=0x7f635c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.287+0000 7f6361e00700 1 -- 172.21.15.104:0/3389523726 shutdown_connections 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.287+0000 7f6361e00700 1 -- 172.21.15.104:0/3389523726 wait complete. 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.288+0000 7f6361e00700 1 Processor -- start 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.288+0000 7f6361e00700 1 -- start start 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.288+0000 7f6361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.288+0000 7f6361e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c119c00 con 0x7f635c0724f0 2024-04-16T07:13:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.289+0000 7f635aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.289+0000 7f635aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:54932/0 (socket says 172.21.15.104:54932) 2024-04-16T07:13:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.289+0000 7f635aa00700 1 -- 172.21.15.104:0/3780174632 learned_addr learned my addr 172.21.15.104:0/3780174632 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.290+0000 7f635aa00700 1 -- 172.21.15.104:0/3780174632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c005040 con 0x7f635c0724f0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.290+0000 7f635aa00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f634c00b820 tx=0x7f634c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.290+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f634c0093c0 con 0x7f635c0724f0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.290+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c119e00 con 0x7f635c0724f0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.291+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f634c013420 con 0x7f635c0724f0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.291+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f634c01c5b0 con 0x7f635c0724f0 2024-04-16T07:13:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.291+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c11a2f0 con 0x7f635c0724f0 2024-04-16T07:13:49.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.292+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c073c30 con 0x7f635c0724f0 2024-04-16T07:13:49.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.292+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f634c013a90 con 0x7f635c0724f0 2024-04-16T07:13:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.293+0000 7f6352a00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 0x7f6340038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.293+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f634c049af0 con 0x7f635c0724f0 2024-04-16T07:13:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.293+0000 7f635a000700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 0x7f6340038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:49.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.294+0000 7f635a000700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 0x7f6340038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6348009990 tx=0x7f6348006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.308+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f634c0098a0 con 0x7f635c0724f0 2024-04-16T07:13:49.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.609+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f635c05f5f0 con 0x7f635c0724f0 2024-04-16T07:13:49.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.610+0000 7f6352a00700 1 -- 172.21.15.104:0/3780174632 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f634c023300 con 0x7f635c0724f0 2024-04-16T07:13:49.609 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:49.609 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:49.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 msgr2=0x7f6340038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:49.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 0x7f6340038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6348009990 tx=0x7f6348006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 msgr2=0x7f635c1196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f634c00b820 tx=0x7f634c009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 shutdown_connections 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f63400369c0 0x7f6340038e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 --2- 172.21.15.104:0/3780174632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f635c0724f0 0x7f635c1196c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 >> 172.21.15.104:0/3780174632 conn(0x7f635c0fa950 msgr2=0x7f635c0fb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 shutdown_connections 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:49.613+0000 7f6361e00700 1 -- 172.21.15.104:0/3780174632 wait complete. 2024-04-16T07:13:49.612 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:50.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:50.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-16T07:13:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:50 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3780174632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:51.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:51.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:51.791 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:52.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:52 smithi026 ceph-mon[28805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:52.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:52 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:54.562 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:54 smithi026 ceph-mon[28805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.228+0000 7f9369000700 1 -- 172.21.15.104:0/3112684930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 msgr2=0x7f9364101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.228+0000 7f9369000700 1 --2- 172.21.15.104:0/3112684930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f9364101db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93540048c0 tx=0x7f9354009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.228+0000 7f9369000700 1 -- 172.21.15.104:0/3112684930 shutdown_connections 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.228+0000 7f9369000700 1 --2- 172.21.15.104:0/3112684930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f9364101db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.228+0000 7f9369000700 1 -- 172.21.15.104:0/3112684930 >> 172.21.15.104:0/3112684930 conn(0x7f93640f7500 msgr2=0x7f93640f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.229+0000 7f9369000700 1 -- 172.21.15.104:0/3112684930 shutdown_connections 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.229+0000 7f9369000700 1 -- 172.21.15.104:0/3112684930 wait complete. 2024-04-16T07:13:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.230+0000 7f9369000700 1 Processor -- start 2024-04-16T07:13:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.230+0000 7f9369000700 1 -- start start 2024-04-16T07:13:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.230+0000 7f9369000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.230+0000 7f9369000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9364066c00 con 0x7f93641019e0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.231+0000 7f9361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.231+0000 7f9361e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:54960/0 (socket says 172.21.15.104:54960) 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.231+0000 7f9361e00700 1 -- 172.21.15.104:0/2201292851 learned_addr learned my addr 172.21.15.104:0/2201292851 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.232+0000 7f9361e00700 1 -- 172.21.15.104:0/2201292851 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9354005040 con 0x7f93641019e0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.232+0000 7f9361e00700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f935400b820 tx=0x7f9354009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.232+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93540093c0 con 0x7f93641019e0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.232+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936410e7e0 con 0x7f93641019e0 2024-04-16T07:13:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.233+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936410ecd0 con 0x7f93641019e0 2024-04-16T07:13:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.233+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9354013420 con 0x7f93641019e0 2024-04-16T07:13:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.233+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93540255c0 con 0x7f93641019e0 2024-04-16T07:13:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.234+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936410ef10 con 0x7f93641019e0 2024-04-16T07:13:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.234+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f9354013590 con 0x7f93641019e0 2024-04-16T07:13:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.234+0000 7f935a000700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 0x7f9348038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.234+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9354049b60 con 0x7f93641019e0 2024-04-16T07:13:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.235+0000 7f9361400700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 0x7f9348038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.236+0000 7f9361400700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 0x7f9348038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9350009990 tx=0x7f9350006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:55.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.250+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f93540238f0 con 0x7f93641019e0 2024-04-16T07:13:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.548+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9364105cd0 con 0x7f93641019e0 2024-04-16T07:13:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.549+0000 7f935a000700 1 -- 172.21.15.104:0/2201292851 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9354011370 con 0x7f93641019e0 2024-04-16T07:13:55.550 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:13:55.550 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 msgr2=0x7f9348038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 0x7f9348038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9350009990 tx=0x7f9350006e40 comp rx=0 tx=0).stop 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 msgr2=0x7f936410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f935400b820 tx=0x7f9354009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 shutdown_connections 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f93480369c0 0x7f9348038e70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 --2- 172.21.15.104:0/2201292851 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93641019e0 0x7f936410e2a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 >> 172.21.15.104:0/2201292851 conn(0x7f93640f7500 msgr2=0x7f93640f8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 shutdown_connections 2024-04-16T07:13:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:55.552+0000 7f9369000700 1 -- 172.21.15.104:0/2201292851 wait complete. 2024-04-16T07:13:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T07:13:56.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:13:56.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:13:56.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:13:56.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/2201292851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:13:56.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:13:57.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:13:57.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:13:57.712 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:13:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:13:58 smithi026 ceph-mon[28805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:13:59.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.973+0000 7fa336e00700 1 -- 172.21.15.104:0/4103585738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 msgr2=0x7fa330070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.973+0000 7fa336e00700 1 --2- 172.21.15.104:0/4103585738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330070730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3240048c0 tx=0x7fa324009e30 comp rx=0 tx=0).stop 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.974+0000 7fa336e00700 1 -- 172.21.15.104:0/4103585738 shutdown_connections 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.974+0000 7fa336e00700 1 --2- 172.21.15.104:0/4103585738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330070730 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.974+0000 7fa336e00700 1 -- 172.21.15.104:0/4103585738 >> 172.21.15.104:0/4103585738 conn(0x7fa3300f83d0 msgr2=0x7fa3300fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.974+0000 7fa336e00700 1 -- 172.21.15.104:0/4103585738 shutdown_connections 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.974+0000 7fa336e00700 1 -- 172.21.15.104:0/4103585738 wait complete. 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.975+0000 7fa336e00700 1 Processor -- start 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.975+0000 7fa336e00700 1 -- start start 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.975+0000 7fa336e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:59.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.975+0000 7fa336e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3301156c0 con 0x7fa330070360 2024-04-16T07:13:59.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.976+0000 7fa335a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:59.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.976+0000 7fa335a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.104:40930/0 (socket says 172.21.15.104:40930) 2024-04-16T07:13:59.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.976+0000 7fa335a00700 1 -- 172.21.15.104:0/3637862896 learned_addr learned my addr 172.21.15.104:0/3637862896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:13:59.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.976+0000 7fa335a00700 1 -- 172.21.15.104:0/3637862896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324005040 con 0x7fa330070360 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa335a00700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b7f0 tx=0x7fa324009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa324009370 con 0x7fa330070360 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3301158c0 con 0x7fa330070360 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa324013420 con 0x7fa330070360 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa32401c590 con 0x7fa330070360 2024-04-16T07:13:59.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.977+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa330115d30 con 0x7fa330070360 2024-04-16T07:13:59.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.979+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330071aa0 con 0x7fa330070360 2024-04-16T07:13:59.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.979+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fa324013a70 con 0x7fa330070360 2024-04-16T07:13:59.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.980+0000 7fa32d600700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 0x7fa318038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:13:59.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.980+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa324049470 con 0x7fa330070360 2024-04-16T07:13:59.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.980+0000 7fa335000700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 0x7fa318038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:13:59.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.981+0000 7fa335000700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 0x7fa318038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009990 tx=0x7fa31c006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:13:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:13:59.997+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa324024d30 con 0x7fa330070360 2024-04-16T07:14:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.291+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa33005f5f0 con 0x7fa330070360 2024-04-16T07:14:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.292+0000 7fa32d600700 1 -- 172.21.15.104:0/3637862896 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa324021070 con 0x7fa330070360 2024-04-16T07:14:00.291 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:14:00.291 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:10:43.696046Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T07:14:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 msgr2=0x7fa318038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 0x7fa318038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009990 tx=0x7fa31c006e40 comp rx=0 tx=0).stop 2024-04-16T07:14:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 msgr2=0x7fa330115180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b7f0 tx=0x7fa324009e30 comp rx=0 tx=0).stop 2024-04-16T07:14:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 shutdown_connections 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa3180369c0 0x7fa318038e70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 --2- 172.21.15.104:0/3637862896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa330070360 0x7fa330115180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 >> 172.21.15.104:0/3637862896 conn(0x7fa3300f83d0 msgr2=0x7fa3301057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 shutdown_connections 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:00.295+0000 7fa336e00700 1 -- 172.21.15.104:0/3637862896 wait complete. 2024-04-16T07:14:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-16T07:14:00.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:00.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:00 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3637862896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:01 smithi026 ceph-mon[28805]: Deploying daemon mon.smithi104 on smithi104 2024-04-16T07:14:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:02.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T07:14:02.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T07:14:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:02 smithi026 ceph-mon[28805]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:03.096 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:04.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:04 smithi026 ceph-mon[28805]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.194 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 podman[36113]: 2024-04-16 07:14:04.981888224 +0000 UTC m=+0.033723976 image pull b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 podman[36113]: 2024-04-16 07:14:05.523282796 +0000 UTC m=+0.575118579 container init dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-64502-quincy-3-45317f9, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 podman[36113]: 2024-04-16 07:14:05.530900762 +0000 UTC m=+0.582736525 container start dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-mon, pid 2 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pidfile_write: ignore empty --pid-file 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: load: jerasure load: lrc 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: RocksDB version: 6.15.5 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-16T07:14:05.723 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Compile date Apr 12 2024 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: DB SUMMARY 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: DB Session ID: 2JRTX888VIRYZ4ATZNL2 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: CURRENT file: CURRENT 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 0, files: 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 000004.log size: 511 ; 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.error_if_exists: 0 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.create_if_missing: 0 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.paranoid_checks: 1 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.env: 0x561ae1285b00 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.fs: Posix File System 2024-04-16T07:14:05.724 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.info_log: 0x561ae9ac7c00 2024-04-16T07:14:05.725 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T07:14:05.725 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.statistics: (nil) 2024-04-16T07:14:05.725 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.use_fsync: 0 2024-04-16T07:14:05.725 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_log_file_size: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_fallocate: 1 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.use_direct_reads: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T07:14:05.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.db_log_dir: 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi104/store.db 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.write_buffer_manager: 0x561aea83ca20 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-16T07:14:05.727 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.unordered_write: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.row_cache: None 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.wal_filter: None 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T07:14:05.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.preserve_deletes: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.two_write_queues: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.atomic_flush: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.log_readahead_size: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T07:14:05.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_background_jobs: 2 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_background_compactions: -1 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_subcompactions: 1 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_open_files: -1 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T07:14:05.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_background_flushes: -1 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Compression algorithms supported: 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kZSTD supported: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kXpressCompression supported: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kZlibCompression supported: 1 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kSnappyCompression supported: 1 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kLZ4Compression supported: 1 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: kBZip2Compression supported: 0 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000003 2024-04-16T07:14:05.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.merge_operator: 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_filter: None 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561ae9b1cf58) 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_type: 0 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_index_type: 0 2024-04-16T07:14:05.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_shortening: 1 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: hash_index_allow_collision: 1 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: checksum: 1 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: no_block_cache: 0 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache: 0x561ae9b4f090 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_options: 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: capacity : 536870912 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_shard_bits : 4 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: strict_capacity_limit : 0 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_compressed: (nil) 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: persistent_cache: (nil) 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size: 4096 2024-04-16T07:14:05.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size_deviation: 10 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_restart_interval: 16 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_block_restart_interval: 1 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_block_size: 4096 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: partition_filters: 0 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: use_delta_encoding: 1 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: whole_key_filtering: 1 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: verify_compression: 0 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: format_version: 4 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_index_compression: 1 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_align: 0 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T07:14:05.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression: NoCompression 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.num_levels: 7 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T07:14:05.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T07:14:05.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.arena_block_size: 4194304 2024-04-16T07:14:05.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T07:14:05.738 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.table_properties_collectors: 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.inplace_update_support: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.bloom_locality: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.max_successive_merges: 0 2024-04-16T07:14:05.739 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.ttl: 2592000 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.enable_blob_files: false 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.min_blob_size: 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251645578998, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-04-16T07:14:05.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251645579566, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713251645, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9adcd018-f940-4d6a-9eae-b81dad9afac0", "db_session_id": "2JRTX888VIRYZ4ATZNL2"}} 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713251645580259, "job": 1, "event": "recovery_finished"} 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi104/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x561ae9ae3180 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: DB pointer 0x561ae9bb8000 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** DB Stats ** 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T07:14:05.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:14:05.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:14:05.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104 does not exist in monmap, will attempt to join an existing cluster 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: starting mon.smithi104 rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi104 fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(???) e0 preinit fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).mds e1 new map 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).mds e1 print_map 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: e1 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: legacy client fscid: -1 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: No filesystems configured 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-16T07:14:05.745 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:14:05.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Deploying daemon crash.smithi104 on smithi104 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/1895106553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/3780174632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:14:05.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/2201292851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/3637862896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: Deploying daemon mon.smithi104 on smithi104 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:05.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 ceph-mon[36163]: mon.smithi104@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-16T07:14:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 bash[36113]: dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 2024-04-16T07:14:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:05 smithi104 systemd[1]: Started Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:14:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:14:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: mon.smithi026 calling monitor election 2024-04-16T07:14:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: mon.smithi104 calling monitor election 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: monmap e2: 2 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: fsmap 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: mgrmap e17: smithi026.tawexg(active, since 39s) 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: overall HEALTH_OK 2024-04-16T07:14:11.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:14:11.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:14:11.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:11.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:14:11.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:14:11.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:10 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:11.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: mon.smithi026 calling monitor election 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: mon.smithi104 calling monitor election 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: monmap e2: 2 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: fsmap 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: mgrmap e17: smithi026.tawexg(active, since 39s) 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: overall HEALTH_OK 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1276168691' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:14:11.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:10 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 -- 172.21.15.104:0/1885207938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89180061f0 msgr2=0x7f8918008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 --2- 172.21.15.104:0/1885207938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89180061f0 0x7f8918008650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8934071420 tx=0x7f8924002040 comp rx=0 tx=0).stop 2024-04-16T07:14:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 -- 172.21.15.104:0/1885207938 shutdown_connections 2024-04-16T07:14:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 --2- 172.21.15.104:0/1885207938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89180061f0 0x7f8918008650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 -- 172.21.15.104:0/1885207938 >> 172.21.15.104:0/1885207938 conn(0x7f89340f83f0 msgr2=0x7f89340fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 -- 172.21.15.104:0/1885207938 shutdown_connections 2024-04-16T07:14:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.774+0000 7f8933200700 1 -- 172.21.15.104:0/1885207938 wait complete. 2024-04-16T07:14:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.775+0000 7f8933200700 1 Processor -- start 2024-04-16T07:14:11.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.776+0000 7f8933200700 1 -- start start 2024-04-16T07:14:11.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.776+0000 7f8933200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:11.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8933200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 0x7f893410f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:11.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8933200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934115e10 con 0x7f8934115800 2024-04-16T07:14:11.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8933200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934115f80 con 0x7f89180061f0 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53276/0 (socket says 172.21.15.104:53276) 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 -- 172.21.15.104:0/4001819233 learned_addr learned my addr 172.21.15.104:0/4001819233 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931400700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 0x7f893410f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 -- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 msgr2=0x7f89341152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 -- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 msgr2=0x7f89341152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.777+0000 7f8931e00700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.778+0000 7f8931e00700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.778+0000 7f8931400700 1 -- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 msgr2=0x7f89341152c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:14:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.778+0000 7f8931400700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.778+0000 7f8931400700 1 -- 172.21.15.104:0/4001819233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924005040 con 0x7f8934115800 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f8931400700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 0x7f893410f260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f892000b790 tx=0x7f89200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8920011810 con 0x7f8934115800 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893410f8c0 con 0x7f8934115800 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8920011e40 con 0x7f8934115800 2024-04-16T07:14:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892000f2e0 con 0x7f8934115800 2024-04-16T07:14:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.779+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893410fe00 con 0x7f8934115800 2024-04-16T07:14:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.781+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89341100d0 con 0x7f8934115800 2024-04-16T07:14:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.783+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f8920010380 con 0x7f8934115800 2024-04-16T07:14:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.784+0000 7f892a000700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 0x7f891406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.784+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8920086420 con 0x7f8934115800 2024-04-16T07:14:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.784+0000 7f8931e00700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 0x7f891406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.785+0000 7f8931e00700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 0x7f891406b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8924004f50 tx=0x7f8924002040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:11.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:11.795+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8920053790 con 0x7f8934115800 2024-04-16T07:14:12.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: mgrmap e18: smithi026.tawexg(active, since 39s), standbys: smithi104.uqfljg 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:11 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: mgrmap e18: smithi026.tawexg(active, since 39s), standbys: smithi104.uqfljg 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:14:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:11 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:14:12.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.104+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89341100d0 con 0x7f8934115800 2024-04-16T07:14:12.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.106+0000 7f892a000700 1 -- 172.21.15.104:0/4001819233 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f89341100d0 con 0x7f8934115800 2024-04-16T07:14:12.105 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-16T07:14:12.105 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","modified":"2024-04-16T07:14:05.634368Z","created":"2024-04-16T07:10:43.696046Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-16T07:14:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 msgr2=0x7f891406b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 0x7f891406b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8924004f50 tx=0x7f8924002040 comp rx=0 tx=0).stop 2024-04-16T07:14:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 msgr2=0x7f893410f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 0x7f893410f260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f892000b790 tx=0x7f89200091a0 comp rx=0 tx=0).stop 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 shutdown_connections 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8914069360 0x7f891406b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89180061f0 0x7f89341152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 --2- 172.21.15.104:0/4001819233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8934115800 0x7f893410f260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 >> 172.21.15.104:0/4001819233 conn(0x7f89340f83f0 msgr2=0x7f8934104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 shutdown_connections 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:14:12.109+0000 7f8933200700 1 -- 172.21.15.104:0/4001819233 wait complete. 2024-04-16T07:14:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-04-16T07:14:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:12 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/4001819233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:12.913 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-16T07:14:12.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-16T07:14:13.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:12 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/4001819233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T07:14:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:14:13.900 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:13 smithi026 ceph-mon[28805]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:14.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:13 smithi104 ceph-mon[36163]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:15.411 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:15.411 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:15.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:15.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:14:15.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:14:15.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:15 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.590+0000 7faadc200700 1 -- 172.21.15.26:0/3660380291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 msgr2=0x7faad40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.590+0000 7faadc200700 1 --2- 172.21.15.26:0/3660380291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad40706d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faac80048c0 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.591+0000 7faadc200700 1 -- 172.21.15.26:0/3660380291 shutdown_connections 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.591+0000 7faadc200700 1 --2- 172.21.15.26:0/3660380291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad40706d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.591+0000 7faadc200700 1 -- 172.21.15.26:0/3660380291 >> 172.21.15.26:0/3660380291 conn(0x7faad40f85e0 msgr2=0x7faad40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.591+0000 7faadc200700 1 -- 172.21.15.26:0/3660380291 shutdown_connections 2024-04-16T07:14:15.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.591+0000 7faadc200700 1 -- 172.21.15.26:0/3660380291 wait complete. 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 Processor -- start 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 -- start start 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4115a40 0x7faad410f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad4116000 con 0x7faad4070300 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.592+0000 7faadc200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad410f9c0 con 0x7faad4115a40 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.593+0000 7faad9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.593+0000 7faad9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54774/0 (socket says 172.21.15.26:54774) 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.593+0000 7faad8e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4115a40 0x7faad410f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:15.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.593+0000 7faad9800700 1 -- 172.21.15.26:0/1821425253 learned_addr learned my addr 172.21.15.26:0/1821425253 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:14:15.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad9800700 1 -- 172.21.15.26:0/1821425253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4115a40 msgr2=0x7faad410f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:15.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad9800700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4115a40 0x7faad410f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad9800700 1 -- 172.21.15.26:0/1821425253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8005040 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad9800700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faac800b5c0 tx=0x7faac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac800e040 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faac80098d0 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac801c3f0 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.594+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad410fc40 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.595+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad4110180 con 0x7faad4070300 2024-04-16T07:14:15.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.596+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad4104b20 con 0x7faad4070300 2024-04-16T07:14:15.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.597+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7faac8013420 con 0x7faad4070300 2024-04-16T07:14:15.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.598+0000 7faad1600700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 0x7faabc06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:15.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.598+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7faac8087360 con 0x7faad4070300 2024-04-16T07:14:15.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.599+0000 7faad8e00700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 0x7faabc06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:15.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.599+0000 7faad8e00700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 0x7faabc06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faac0009b80 tx=0x7faac00091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:15.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.606+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faac8058290 con 0x7faad4070300 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:14:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:15 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:14:15.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.839+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7faad405f650 con 0x7faad4070300 2024-04-16T07:14:15.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.840+0000 7faad1600700 1 -- 172.21.15.26:0/1821425253 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7faac8057e20 con 0x7faad4070300 2024-04-16T07:14:15.839 INFO:teuthology.orchestra.run.smithi026.stdout:# minimal ceph.conf for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:14:15.839 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-04-16T07:14:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:14:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: mon_host = [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-04-16T07:14:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 msgr2=0x7faabc06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 0x7faabc06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faac0009b80 tx=0x7faac00091a0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 msgr2=0x7faad4115500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faac800b5c0 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 shutdown_connections 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7faabc069360 0x7faabc06b810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faad4070300 0x7faad4115500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 --2- 172.21.15.26:0/1821425253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4115a40 0x7faad410f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.843+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 >> 172.21.15.26:0/1821425253 conn(0x7faad40f85e0 msgr2=0x7faad40fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.844+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 shutdown_connections 2024-04-16T07:14:15.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:15.844+0000 7faadc200700 1 -- 172.21.15.26:0/1821425253 wait complete. 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1821425253' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:16.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:16.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:14:16.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:16.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:16.731 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-16T07:14:16.731 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:14:16.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T07:14:16.815 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:14:16.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:14:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:14:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1821425253' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:16.951 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:14:16.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T07:14:17.055 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:14:17.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:14:17.185 INFO:tasks.cephadm:Deploying OSDs... 2024-04-16T07:14:17.186 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:14:17.186 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:14:17.247 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:14:17.247 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 625 Links: 1 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:17.303 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 07:13:20.247137332 +0000 2024-04-16T07:14:17.304 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 07:06:23.448323369 +0000 2024-04-16T07:14:17.304 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 07:06:23.448323369 +0000 2024-04-16T07:14:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 07:06:23.448323369 +0000 2024-04-16T07:14:17.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:14:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T07:14:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T07:14:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000196589 s, 2.6 MB/s 2024-04-16T07:14:17.385 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:14:17.433 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 647 Links: 1 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 07:13:20.247137332 +0000 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 07:06:24.063313643 +0000 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 07:06:24.063313643 +0000 2024-04-16T07:14:17.482 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 07:06:24.063313643 +0000 2024-04-16T07:14:17.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:14:17.567 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:17 smithi026 ceph-mon[28805]: Reconfiguring mon.smithi026 (unknown last config time)... 2024-04-16T07:14:17.567 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:17 smithi026 ceph-mon[28805]: Reconfiguring daemon mon.smithi026 on smithi026 2024-04-16T07:14:17.582 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T07:14:17.582 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T07:14:17.582 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00026852 s, 1.9 MB/s 2024-04-16T07:14:17.587 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:14:17.635 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 665 Links: 1 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 07:13:20.248137316 +0000 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 07:06:24.665304123 +0000 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 07:06:24.665304123 +0000 2024-04-16T07:14:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 07:06:24.665304123 +0000 2024-04-16T07:14:17.681 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:14:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T07:14:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T07:14:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000257558 s, 2.0 MB/s 2024-04-16T07:14:17.761 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:14:17.815 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-04-16T07:14:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:17 smithi104 ceph-mon[36163]: Reconfiguring mon.smithi026 (unknown last config time)... 2024-04-16T07:14:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:17 smithi104 ceph-mon[36163]: Reconfiguring daemon mon.smithi026 on smithi026 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 684 Links: 1 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 07:13:20.249137301 +0000 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 07:06:25.273294507 +0000 2024-04-16T07:14:17.867 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 07:06:25.273294507 +0000 2024-04-16T07:14:17.868 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 07:06:25.273294507 +0000 2024-04-16T07:14:17.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:14:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T07:14:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T07:14:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000265112 s, 1.9 MB/s 2024-04-16T07:14:17.966 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:14:18.027 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:14:18.027 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:14:18.087 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:14:18.087 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-16 07:14:03.990106873 +0000 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-16 07:06:11.795151336 +0000 2024-04-16T07:14:18.139 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-16 07:06:11.795151336 +0000 2024-04-16T07:14:18.140 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-16 07:06:11.795151336 +0000 2024-04-16T07:14:18.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:14:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-16T07:14:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-16T07:14:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000427935 s, 1.2 MB/s 2024-04-16T07:14:18.235 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:14:18.294 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-04-16T07:14:18.344 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:14:18.344 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:18.344 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-16 07:14:03.990106873 +0000 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-16 07:06:12.402144125 +0000 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-16 07:06:12.402144125 +0000 2024-04-16T07:14:18.345 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-16 07:06:12.402144125 +0000 2024-04-16T07:14:18.345 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:14:18.430 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-16T07:14:18.430 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-16T07:14:18.430 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000327334 s, 1.6 MB/s 2024-04-16T07:14:18.433 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:14:18.487 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-04-16T07:14:18.535 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-16 07:14:03.991106862 +0000 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-16 07:06:13.027136700 +0000 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-16 07:06:13.027136700 +0000 2024-04-16T07:14:18.536 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-16 07:06:13.027136700 +0000 2024-04-16T07:14:18.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:14:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:18.618 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-16T07:14:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-16T07:14:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000345718 s, 1.5 MB/s 2024-04-16T07:14:18.622 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:14:18.722 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:14:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:18.773 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-16 07:14:03.992106850 +0000 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-16 07:06:13.622129632 +0000 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-16 07:06:13.622129632 +0000 2024-04-16T07:14:18.774 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-16 07:06:13.622129632 +0000 2024-04-16T07:14:18.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:14:18.864 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-16T07:14:18.864 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-16T07:14:18.865 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000341711 s, 1.5 MB/s 2024-04-16T07:14:18.868 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:14:18.923 INFO:tasks.cephadm:Deploying osd.0 on smithi026 with /dev/vg_nvme/lv_4... 2024-04-16T07:14:18.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T07:14:19.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:19 smithi026 ceph-mon[28805]: Reconfiguring mgr.smithi026.tawexg (unknown last config time)... 2024-04-16T07:14:19.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:19 smithi026 ceph-mon[28805]: Reconfiguring daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:14:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:19 smithi104 ceph-mon[36163]: Reconfiguring mgr.smithi026.tawexg (unknown last config time)... 2024-04-16T07:14:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:19 smithi104 ceph-mon[36163]: Reconfiguring daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:14:20.224 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:14:20.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:20 smithi026 ceph-mon[28805]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:20.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:20.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:20.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:14:20.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:20 smithi104 ceph-mon[36163]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:14:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:21.435 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:21 smithi026 ceph-mon[28805]: Reconfiguring crash.smithi026 (monmap changed)... 2024-04-16T07:14:21.435 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:21 smithi026 ceph-mon[28805]: Reconfiguring daemon crash.smithi026 on smithi026 2024-04-16T07:14:21.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:21 smithi104 ceph-mon[36163]: Reconfiguring crash.smithi026 (monmap changed)... 2024-04-16T07:14:21.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:21 smithi104 ceph-mon[36163]: Reconfiguring daemon crash.smithi026 on smithi026 2024-04-16T07:14:22.643 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:22 smithi026 ceph-mon[28805]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:22 smithi104 ceph-mon[36163]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0414062 s, 253 MB/s 2024-04-16T07:14:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-16T07:14:23.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_4 2024-04-16T07:14:24.481 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:24.482 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:24.482 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:24 smithi026 ceph-mon[28805]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-04-16T07:14:24.482 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:24 smithi026 ceph-mon[28805]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-04-16T07:14:24.482 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:24 smithi026 ceph-mon[28805]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:24 smithi104 ceph-mon[36163]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-04-16T07:14:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:24 smithi104 ceph-mon[36163]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-04-16T07:14:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:24 smithi104 ceph-mon[36163]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:25.711 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:14:26.339 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:26 smithi026 ceph-mon[28805]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:26 smithi104 ceph-mon[36163]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:28.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:28 smithi026 ceph-mon[28805]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:28.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:28 smithi104 ceph-mon[36163]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 -- 172.21.15.26:0/3407997421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 msgr2=0x7f66f00fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 --2- 172.21.15.26:0/3407997421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f00fe070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f66dc004940 tx=0x7f66dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:14:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 -- 172.21.15.26:0/3407997421 shutdown_connections 2024-04-16T07:14:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 --2- 172.21.15.26:0/3407997421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f00fe070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 --2- 172.21.15.26:0/3407997421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 0x7f66f00fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.093+0000 7f66f5e00700 1 -- 172.21.15.26:0/3407997421 >> 172.21.15.26:0/3407997421 conn(0x7f66f00f8540 msgr2=0x7f66f00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:14:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.094+0000 7f66f5e00700 1 -- 172.21.15.26:0/3407997421 shutdown_connections 2024-04-16T07:14:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.094+0000 7f66f5e00700 1 -- 172.21.15.26:0/3407997421 wait complete. 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.095+0000 7f66f5e00700 1 Processor -- start 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.095+0000 7f66f5e00700 1 -- start start 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.095+0000 7f66f5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 0x7f66f010fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66f5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66f5e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0110890 con 0x7f66f00fd710 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66f5e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0110fd0 con 0x7f66f00fcdc0 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53236/0 (socket says 172.21.15.26:53236) 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 -- 172.21.15.26:0/670285506 learned_addr learned my addr 172.21.15.26:0/670285506 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66eea00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 0x7f66f010fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 -- 172.21.15.26:0/670285506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 msgr2=0x7f66f010fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 0x7f66f010fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.096+0000 7f66ee000700 1 -- 172.21.15.26:0/670285506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc005040 con 0x7f66f00fd710 2024-04-16T07:14:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66ee000700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00b970 tx=0x7f66dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66dc00e040 con 0x7f66f00fd710 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66dc004290 con 0x7f66f00fd710 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66dc01c630 con 0x7f66f00fd710 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f0111250 con 0x7f66f00fd710 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.097+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f0111810 con 0x7f66f00fd710 2024-04-16T07:14:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.099+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f0101a60 con 0x7f66f00fd710 2024-04-16T07:14:29.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.100+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f66dc00bbf0 con 0x7f66f00fd710 2024-04-16T07:14:29.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.101+0000 7f66e6a00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 0x7f66d406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:14:29.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.101+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f66dc087400 con 0x7f66f00fd710 2024-04-16T07:14:29.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.101+0000 7f66eea00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 0x7f66d406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:14:29.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.102+0000 7f66eea00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 0x7f66d406b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009fd0 tx=0x7f66e0009f00 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:14:29.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.106+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f66dc058280 con 0x7f66f00fd710 2024-04-16T07:14:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:14:29.333+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f66f0105a90 con 0x7f66d4069210 2024-04-16T07:14:30.345 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:30 smithi026 ceph-mon[28805]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:30.345 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:30 smithi026 ceph-mon[28805]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:14:30.345 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:14:30.345 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:14:30.345 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:30 smithi104 ceph-mon[36163]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:30 smithi104 ceph-mon[36163]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:14:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:14:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:14:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:31.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:31.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:32 smithi104 ceph-mon[36163]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:32.673 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:32 smithi026 ceph-mon[28805]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:32.673 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:32.673 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:33.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:33 smithi104 ceph-mon[36163]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-04-16T07:14:33.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:33 smithi104 ceph-mon[36163]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-04-16T07:14:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:33 smithi026 ceph-mon[28805]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-04-16T07:14:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:33 smithi026 ceph-mon[28805]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-04-16T07:14:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:34 smithi104 ceph-mon[36163]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:34 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/615641447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302d9de6-c04f-40d5-af51-575b9d1cc85b"}]: dispatch 2024-04-16T07:14:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:34 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/615641447' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302d9de6-c04f-40d5-af51-575b9d1cc85b"}]': finished 2024-04-16T07:14:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:34 smithi104 ceph-mon[36163]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T07:14:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:34 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:14:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:34 smithi026 ceph-mon[28805]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:34 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/615641447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302d9de6-c04f-40d5-af51-575b9d1cc85b"}]: dispatch 2024-04-16T07:14:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:34 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/615641447' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302d9de6-c04f-40d5-af51-575b9d1cc85b"}]': finished 2024-04-16T07:14:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:34 smithi026 ceph-mon[28805]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T07:14:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:34 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:14:35.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:35 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/859841314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:14:35.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:35 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/859841314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:14:36.711 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:36 smithi026 ceph-mon[28805]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:36 smithi104 ceph-mon[36163]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:38.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:38 smithi026 ceph-mon[28805]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:38 smithi104 ceph-mon[36163]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:40.680 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:40 smithi026 ceph-mon[28805]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:40 smithi104 ceph-mon[36163]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:42.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:42 smithi026 ceph-mon[28805]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:42 smithi104 ceph-mon[36163]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:44.615 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:44 smithi026 ceph-mon[28805]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:44.634 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:44 smithi104 ceph-mon[36163]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:46.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:46 smithi104 ceph-mon[36163]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:46.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:46.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:46.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:46.695 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:46 smithi026 ceph-mon[28805]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:46.695 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:46.695 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:46.696 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:14:47.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:47 smithi104 ceph-mon[36163]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:14:47.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:47 smithi104 ceph-mon[36163]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:14:47.749 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:47 smithi026 ceph-mon[28805]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:14:47.749 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:47 smithi026 ceph-mon[28805]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:14:48.493 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:48 smithi026 ceph-mon[28805]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:48.494 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:14:48.494 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:48.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:48 smithi104 ceph-mon[36163]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:48.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:14:48.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:49.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:49 smithi026 ceph-mon[28805]: Deploying daemon osd.0 on smithi026 2024-04-16T07:14:49.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:49 smithi104 ceph-mon[36163]: Deploying daemon osd.0 on smithi026 2024-04-16T07:14:50.493 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:50 smithi026 ceph-mon[28805]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:50 smithi104 ceph-mon[36163]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:52 smithi026 ceph-mon[28805]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:52 smithi104 ceph-mon[36163]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:54 smithi026 ceph-mon[28805]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-16T07:14:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-16T07:14:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:14:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:54 smithi104 ceph-mon[36163]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: Reconfiguring mgr.smithi104.uqfljg (monmap changed)... 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:56.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:56 smithi026 ceph-mon[28805]: Reconfiguring daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:14:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: Reconfiguring mgr.smithi104.uqfljg (monmap changed)... 2024-04-16T07:14:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:14:56.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:14:56.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:56.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:56 smithi104 ceph-mon[36163]: Reconfiguring daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:14:58 smithi026 ceph-mon[28805]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-16T07:14:58.803 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:14:58.803 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:58.803 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:14:58.803 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-16T07:14:58.803 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:14:58.804 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:14:58.804 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:14:58.804 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:14:58 smithi104 ceph-mon[36163]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-16T07:15:00.408 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.408 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T07:15:00.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T07:15:00.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:00.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:02.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:02.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:04.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:04 smithi026 ceph-mon[28805]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:04 smithi104 ceph-mon[36163]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:05.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:05 smithi026 ceph-mon[28805]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T07:15:05.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:05 smithi104 ceph-mon[36163]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T07:15:06.445 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:06 smithi026 ceph-mon[28805]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:06.445 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:06 smithi026 ceph-mon[28805]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T07:15:06.445 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:06 smithi026 ceph-mon[28805]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T07:15:06.445 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:06 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:06.445 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:06 smithi026 ceph-mon[28805]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:15:06.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.556+0000 7f66e6a00700 1 -- 172.21.15.26:0/670285506 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f66e0004630 con 0x7f66d4069210 2024-04-16T07:15:06.555 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 0 on host 'smithi026' 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 msgr2=0x7f66d406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 0x7f66d406b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009fd0 tx=0x7f66e0009f00 comp rx=0 tx=0).stop 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 msgr2=0x7f66f0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00b970 tx=0x7f66dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 shutdown_connections 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f66d4069210 0x7f66d406b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66f00fcdc0 0x7f66f010fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 --2- 172.21.15.26:0/670285506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66f00fd710 0x7f66f0110290 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:06.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 >> 172.21.15.26:0/670285506 conn(0x7f66f00f8540 msgr2=0x7f66f00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:15:06.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 shutdown_connections 2024-04-16T07:15:06.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:06.560+0000 7f66f5e00700 1 -- 172.21.15.26:0/670285506 wait complete. 2024-04-16T07:15:06.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:06 smithi104 ceph-mon[36163]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:06.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:06 smithi104 ceph-mon[36163]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T07:15:06.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:06 smithi104 ceph-mon[36163]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T07:15:06.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:06 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:06.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:06 smithi104 ceph-mon[36163]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:15:07.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:15:07.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T07:15:07.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:07.751 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: from='osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:15:07.751 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T07:15:07.751 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:07.751 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:07.751 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:07.752 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:07.839 DEBUG:teuthology.orchestra.run.smithi026:osd.0> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.0.service 2024-04-16T07:15:07.845 INFO:tasks.cephadm:Deploying osd.1 on smithi026 with /dev/vg_nvme/lv_3... 2024-04-16T07:15:07.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T07:15:08.376 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:15:08.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] boot 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:08.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: osd.0 [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] boot 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:10.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:09 smithi026 ceph-mon[28805]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T07:15:10.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:09 smithi026 ceph-mon[28805]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:10.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:09 smithi104 ceph-mon[36163]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T07:15:10.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:09 smithi104 ceph-mon[36163]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0329299 s, 318 MB/s 2024-04-16T07:15:11.494 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-16T07:15:12.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_3 2024-04-16T07:15:12.577 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:12 smithi026 ceph-mon[28805]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:12.581 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:12 smithi104 ceph-mon[36163]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:12.931 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:15:14.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:14 smithi026 ceph-mon[28805]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:14.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:14 smithi104 ceph-mon[36163]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.764+0000 7fd347200700 1 -- 172.21.15.26:0/1989183089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348071e40 msgr2=0x7fd348070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.764+0000 7fd347200700 1 --2- 172.21.15.26:0/1989183089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348071e40 0x7fd348070350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd338004940 tx=0x7fd338009e30 comp rx=0 tx=0).stop 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 -- 172.21.15.26:0/1989183089 shutdown_connections 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 --2- 172.21.15.26:0/1989183089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070890 0x7fd348070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 --2- 172.21.15.26:0/1989183089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348071e40 0x7fd348070350 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 -- 172.21.15.26:0/1989183089 >> 172.21.15.26:0/1989183089 conn(0x7fd3480f83d0 msgr2=0x7fd3480fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 -- 172.21.15.26:0/1989183089 shutdown_connections 2024-04-16T07:15:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.765+0000 7fd347200700 1 -- 172.21.15.26:0/1989183089 wait complete. 2024-04-16T07:15:15.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 Processor -- start 2024-04-16T07:15:15.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 -- start start 2024-04-16T07:15:15.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:15.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348071e40 0x7fd34810ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:15.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348110630 con 0x7fd348070890 2024-04-16T07:15:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd347200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348110e20 con 0x7fd348071e40 2024-04-16T07:15:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.766+0000 7fd345e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39482/0 (socket says 172.21.15.26:39482) 2024-04-16T07:15:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 -- 172.21.15.26:0/2770859387 learned_addr learned my addr 172.21.15.26:0/2770859387 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:15:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345400700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348071e40 0x7fd34810ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 -- 172.21.15.26:0/2770859387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348071e40 msgr2=0x7fd34810ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348071e40 0x7fd34810ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 -- 172.21.15.26:0/2770859387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338005040 con 0x7fd348070890 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.767+0000 7fd345e00700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b970 tx=0x7fd338009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.768+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd33800e040 con 0x7fd348070890 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.768+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd338004290 con 0x7fd348070890 2024-04-16T07:15:15.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.768+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3481110a0 con 0x7fd348070890 2024-04-16T07:15:15.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.768+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd33801c3f0 con 0x7fd348070890 2024-04-16T07:15:15.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.768+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd348111660 con 0x7fd348070890 2024-04-16T07:15:15.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.769+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34805f5f0 con 0x7fd348070890 2024-04-16T07:15:15.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.771+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fd33800bbf0 con 0x7fd348070890 2024-04-16T07:15:15.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.772+0000 7fd33e000700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 0x7fd32c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:15.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.772+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fd338087560 con 0x7fd348070890 2024-04-16T07:15:15.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.772+0000 7fd345400700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 0x7fd32c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:15.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.773+0000 7fd345400700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 0x7fd32c06b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd334009bf0 tx=0x7fd3340091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:15:15.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:15.777+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd338058230 con 0x7fd348070890 2024-04-16T07:15:16.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:16.004+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd348111a50 con 0x7fd32c069210 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi026 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi026 to 12870M 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:15:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:15:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:16.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi026 2024-04-16T07:15:16.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:16.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi026 to 12870M 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:15:16.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:17.459 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:17 smithi104 ceph-mon[36163]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:15:17.459 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:17 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:17.492 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:17 smithi026 ceph-mon[28805]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:15:17.492 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:17 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:18.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:18 smithi026 ceph-mon[28805]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:18 smithi104 ceph-mon[36163]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:20 smithi104 ceph-mon[36163]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:20.716 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:20 smithi026 ceph-mon[28805]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:21.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:21 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3780208691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ddb8857-f53a-425e-982e-7f2c90978ccc"}]: dispatch 2024-04-16T07:15:21.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:21 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3780208691' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ddb8857-f53a-425e-982e-7f2c90978ccc"}]': finished 2024-04-16T07:15:21.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:21 smithi026 ceph-mon[28805]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T07:15:21.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:21 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:21.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:21 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3780208691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ddb8857-f53a-425e-982e-7f2c90978ccc"}]: dispatch 2024-04-16T07:15:21.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:21 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3780208691' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ddb8857-f53a-425e-982e-7f2c90978ccc"}]': finished 2024-04-16T07:15:21.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:21 smithi104 ceph-mon[36163]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T07:15:21.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:21 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2238135442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:22.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2238135442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:15:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:24.564 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:24 smithi026 ceph-mon[28805]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:24 smithi104 ceph-mon[36163]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:26 smithi104 ceph-mon[36163]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:26.597 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:26 smithi026 ceph-mon[28805]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:28.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:28 smithi104 ceph-mon[36163]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:28 smithi026 ceph-mon[28805]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:30 smithi104 ceph-mon[36163]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:15:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:30 smithi026 ceph-mon[28805]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:15:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:31.727 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:31 smithi026 ceph-mon[28805]: Deploying daemon osd.1 on smithi026 2024-04-16T07:15:31.727 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:31 smithi104 ceph-mon[36163]: Deploying daemon osd.1 on smithi026 2024-04-16T07:15:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:32.491 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:32 smithi026 ceph-mon[28805]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:32 smithi104 ceph-mon[36163]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:34.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:34 smithi026 ceph-mon[28805]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:34 smithi104 ceph-mon[36163]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:36.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:36 smithi026 ceph-mon[28805]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:36 smithi104 ceph-mon[36163]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:38.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:38 smithi026 ceph-mon[28805]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:38.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:38.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:38.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:38 smithi104 ceph-mon[36163]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:40.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:40 smithi026 ceph-mon[28805]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:40 smithi104 ceph-mon[36163]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:41 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:41 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:41.854 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:41 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:41.854 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:41 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:42 smithi026 ceph-mon[28805]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:42 smithi026 ceph-mon[28805]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T07:15:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:42 smithi104 ceph-mon[36163]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:42 smithi104 ceph-mon[36163]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T07:15:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.051+0000 7fd33e000700 1 -- 172.21.15.26:0/2770859387 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd348111a50 con 0x7fd32c069210 2024-04-16T07:15:43.050 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 1 on host 'smithi026' 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 msgr2=0x7fd32c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 0x7fd32c06b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd334009bf0 tx=0x7fd3340091a0 comp rx=0 tx=0).stop 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 msgr2=0x7fd34810fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd33800b970 tx=0x7fd338009e30 comp rx=0 tx=0).stop 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 shutdown_connections 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd32c069210 0x7fd32c06b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd348070890 0x7fd34810fa60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 --2- 172.21.15.26:0/2770859387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348071e40 0x7fd34810ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 >> 172.21.15.26:0/2770859387 conn(0x7fd3480f83d0 msgr2=0x7fd3480f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:15:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 shutdown_connections 2024-04-16T07:15:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:43.055+0000 7fd347200700 1 -- 172.21.15.26:0/2770859387 wait complete. 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:43 smithi026 ceph-mon[28805]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:43.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T07:15:43.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T07:15:43.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:43.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:15:43.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:43.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:43.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:43 smithi104 ceph-mon[36163]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:44.462 DEBUG:teuthology.orchestra.run.smithi026:osd.1> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.1.service 2024-04-16T07:15:44.469 INFO:tasks.cephadm:Deploying osd.2 on smithi026 with /dev/vg_nvme/lv_2... 2024-04-16T07:15:44.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T07:15:44.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:44 smithi026 ceph-mon[28805]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:15:44.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:44 smithi026 ceph-mon[28805]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T07:15:44.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:44.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:44 smithi104 ceph-mon[36163]: from='osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:15:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:44 smithi104 ceph-mon[36163]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T07:15:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:15:45.828 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:15:45.828 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:15:45.829 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] boot 2024-04-16T07:15:45.829 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T07:15:45.829 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:45.829 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:45 smithi026 ceph-mon[28805]: pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:45.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:15:45.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:15:45.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: osd.1 [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] boot 2024-04-16T07:15:45.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T07:15:45.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:15:45.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:45 smithi104 ceph-mon[36163]: pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T07:15:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:46 smithi104 ceph-mon[36163]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T07:15:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:47.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:46 smithi026 ceph-mon[28805]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T07:15:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:15:47.846 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:47 smithi026 ceph-mon[28805]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:48.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:47 smithi104 ceph-mon[36163]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:48.701 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T07:15:48.702 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T07:15:48.702 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-16T07:15:48.702 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-16T07:15:48.702 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0353647 s, 297 MB/s 2024-04-16T07:15:48.702 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-16T07:15:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:49.342 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:49.342 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:50.211 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:50 smithi026 ceph-mon[28805]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:50.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_2 2024-04-16T07:15:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:50 smithi104 ceph-mon[36163]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:50.878 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:15:52.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:52 smithi104 ceph-mon[36163]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:52.749 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:52 smithi026 ceph-mon[28805]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 -- 172.21.15.26:0/2796766850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 msgr2=0x7fb96c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 --2- 172.21.15.26:0/2796766850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c0fe070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb958004940 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 -- 172.21.15.26:0/2796766850 shutdown_connections 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 --2- 172.21.15.26:0/2796766850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c0fe070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 --2- 172.21.15.26:0/2796766850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 -- 172.21.15.26:0/2796766850 >> 172.21.15.26:0/2796766850 conn(0x7fb96c0f8540 msgr2=0x7fb96c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 -- 172.21.15.26:0/2796766850 shutdown_connections 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.843+0000 7fb972c00700 1 -- 172.21.15.26:0/2796766850 wait complete. 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 Processor -- start 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 -- start start 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c10fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c1108a0 con 0x7fb96c0fd710 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.845+0000 7fb972c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c110fe0 con 0x7fb96c0fcdc0 2024-04-16T07:15:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43780/0 (socket says 172.21.15.26:43780) 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 -- 172.21.15.26:0/745033864 learned_addr learned my addr 172.21.15.26:0/745033864 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96be00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c10fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 -- 172.21.15.26:0/745033864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 msgr2=0x7fb96c10fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c10fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96b400700 1 -- 172.21.15.26:0/745033864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958005040 con 0x7fb96c0fd710 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.846+0000 7fb96be00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb96b400700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b970 tx=0x7fb958009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95800e040 con 0x7fb96c0fd710 2024-04-16T07:15:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb958004290 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95801c630 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c111260 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.847+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c111820 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.848+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c05f5f0 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.849+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fb95800bbf0 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.850+0000 7fb963e00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 0x7fb95006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.850+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fb958087810 con 0x7fb96c0fd710 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.850+0000 7fb96be00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 0x7fb95006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.851+0000 7fb96be00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 0x7fb95006b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0041d0 tx=0x7fb95c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:15:54.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:53.856+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb958058280 con 0x7fb96c0fd710 2024-04-16T07:15:54.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:15:54.099+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb96c111b50 con 0x7fb950069210 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:54.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:15:54.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:15:54.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:15:54.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:15:55.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:55 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi026 2024-04-16T07:15:55.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:55 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi026 to 6435M 2024-04-16T07:15:55.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:55 smithi026 ceph-mon[28805]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:15:55.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:55 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:55.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:55 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi026 2024-04-16T07:15:55.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:55 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi026 to 6435M 2024-04-16T07:15:55.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:55 smithi104 ceph-mon[36163]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:15:55.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:55 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:15:56.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:56 smithi104 ceph-mon[36163]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:56.590 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:56 smithi026 ceph-mon[28805]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:58.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:58 smithi026 ceph-mon[28805]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:58.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1702162916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef"}]: dispatch 2024-04-16T07:15:58.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1702162916' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef"}]': finished 2024-04-16T07:15:58.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:58 smithi026 ceph-mon[28805]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T07:15:58.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:15:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:58 smithi104 ceph-mon[36163]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:15:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:58 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1702162916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef"}]: dispatch 2024-04-16T07:15:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:58 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1702162916' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef"}]': finished 2024-04-16T07:15:58.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:58 smithi104 ceph-mon[36163]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T07:15:58.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:15:59.749 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:15:59 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3920251828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:15:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:15:59 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3920251828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:16:00.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:00 smithi026 ceph-mon[28805]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:00 smithi104 ceph-mon[36163]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:04.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:04 smithi026 ceph-mon[28805]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:04 smithi104 ceph-mon[36163]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:06.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:06 smithi026 ceph-mon[28805]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:06.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:06 smithi104 ceph-mon[36163]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:07.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:16:07.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:07 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:07.834 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:16:07.834 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:07 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:08.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:08 smithi026 ceph-mon[28805]: Deploying daemon osd.2 on smithi026 2024-04-16T07:16:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:08 smithi026 ceph-mon[28805]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:08 smithi104 ceph-mon[36163]: Deploying daemon osd.2 on smithi026 2024-04-16T07:16:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:08 smithi104 ceph-mon[36163]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:10.713 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:10 smithi026 ceph-mon[28805]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:10 smithi104 ceph-mon[36163]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:12 smithi104 ceph-mon[36163]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:12.854 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:12 smithi026 ceph-mon[28805]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:14 smithi104 ceph-mon[36163]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:14.923 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:14 smithi026 ceph-mon[28805]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:16.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:16 smithi026 ceph-mon[28805]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:16.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:16.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:16.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:16.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:16 smithi104 ceph-mon[36163]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:18 smithi026 ceph-mon[28805]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:18 smithi104 ceph-mon[36163]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:19.646 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:19 smithi026 ceph-mon[28805]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T07:16:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:19 smithi104 ceph-mon[36163]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T07:16:20.134 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 2 on host 'smithi026' 2024-04-16T07:16:20.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.134+0000 7fb963e00700 1 -- 172.21.15.26:0/745033864 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb96c111b50 con 0x7fb950069210 2024-04-16T07:16:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 msgr2=0x7fb95006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 0x7fb95006b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0041d0 tx=0x7fb95c009040 comp rx=0 tx=0).stop 2024-04-16T07:16:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 msgr2=0x7fb96c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b970 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-04-16T07:16:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 shutdown_connections 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb950069210 0x7fb95006b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c0fcdc0 0x7fb96c10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 --2- 172.21.15.26:0/745033864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb96c0fd710 0x7fb96c1102a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.137+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 >> 172.21.15.26:0/745033864 conn(0x7fb96c0f8540 msgr2=0x7fb96c0f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.138+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 shutdown_connections 2024-04-16T07:16:20.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:16:20.138+0000 7fb972c00700 1 -- 172.21.15.26:0/745033864 wait complete. 2024-04-16T07:16:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:21.646 DEBUG:teuthology.orchestra.run.smithi026:osd.2> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.2.service 2024-04-16T07:16:21.651 INFO:tasks.cephadm:Deploying osd.3 on smithi104 with /dev/vg_nvme/lv_4... 2024-04-16T07:16:21.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T07:16:21.718 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:21 smithi026 ceph-mon[28805]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:16:21.718 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:21 smithi026 ceph-mon[28805]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T07:16:21.718 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:21 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:21.719 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:21 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:21.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:21 smithi104 ceph-mon[36163]: from='osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-16T07:16:21.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:21 smithi104 ceph-mon[36163]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T07:16:21.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:21 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:21.741 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:21 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:22.113 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:16:22.580 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:16:22.580 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:16:22.580 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: pgmap v91: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:22.580 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] boot 2024-04-16T07:16:22.580 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T07:16:22.581 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:22 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: pgmap v91: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: osd.2 [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] boot 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T07:16:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:22 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:16:24.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:23 smithi026 ceph-mon[28805]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T07:16:24.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:23 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T07:16:24.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:23 smithi104 ceph-mon[36163]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T07:16:24.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:23 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0357848 s, 293 MB/s 2024-04-16T07:16:24.978 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-16T07:16:25.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: pgmap v94: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:25.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:25.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:25.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T07:16:25.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T07:16:25.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T07:16:25.273 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: pgmap v94: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:25.273 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:25.273 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:25.273 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T07:16:25.274 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T07:16:25.274 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T07:16:25.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-04-16T07:16:26.164 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:16:26.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T07:16:26.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:25 smithi026 ceph-mon[28805]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T07:16:26.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:25 smithi026 ceph-mon[28805]: pgmap v97: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:26.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63789]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T07:16:26.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63789]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:26.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63789]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:26.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63789]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:26.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T07:16:26.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:25 smithi104 ceph-mon[36163]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T07:16:26.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:25 smithi104 ceph-mon[36163]: pgmap v97: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:26.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63808]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T07:16:26.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:26.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63808]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:26.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:16:26 smithi026 sudo[63808]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:27.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:26 smithi026 ceph-mon[28805]: osdmap e23: 3 total, 3 up, 3 in 2024-04-16T07:16:27.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:26 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:27.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:27 smithi104 ceph-mon[36163]: osdmap e23: 3 total, 3 up, 3 in 2024-04-16T07:16:27.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:27 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:27.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63832]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T07:16:27.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63832]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:27.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63832]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:27.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63832]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:28.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:28 smithi104 ceph-mon[36163]: pgmap v99: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:28.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:28 smithi026 ceph-mon[28805]: pgmap v99: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:28.260 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63844]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T07:16:28.260 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:28.260 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63844]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:28.260 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:16:27 smithi026 sudo[63844]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 -- 172.21.15.104:0/1917647835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc073170 msgr2=0x7f2cfc073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 --2- 172.21.15.104:0/1917647835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc073170 0x7f2cfc073b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4004940 tx=0x7f2ce4009e30 comp rx=0 tx=0).stop 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 -- 172.21.15.104:0/1917647835 shutdown_connections 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 --2- 172.21.15.104:0/1917647835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc073170 0x7f2cfc073b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 --2- 172.21.15.104:0/1917647835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc072820 0x7f2cfc072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 -- 172.21.15.104:0/1917647835 >> 172.21.15.104:0/1917647835 conn(0x7f2cfc0fa950 msgr2=0x7f2cfc0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:16:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 -- 172.21.15.104:0/1917647835 shutdown_connections 2024-04-16T07:16:28.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.289+0000 7f2d03200700 1 -- 172.21.15.104:0/1917647835 wait complete. 2024-04-16T07:16:28.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 Processor -- start 2024-04-16T07:16:28.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 -- start start 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc114480 0x7f2cfc1194b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc114940 con 0x7f2cfc114480 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2d03200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc114a80 con 0x7f2cfc072820 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2cfbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2cfbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45440/0 (socket says 172.21.15.104:45440) 2024-04-16T07:16:28.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.290+0000 7f2cfbe00700 1 -- 172.21.15.104:0/321795100 learned_addr learned my addr 172.21.15.104:0/321795100 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cf3e00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc114480 0x7f2cfc1194b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cfbe00700 1 -- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc114480 msgr2=0x7f2cfc1194b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cfbe00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc114480 0x7f2cfc1194b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cfbe00700 1 -- 172.21.15.104:0/321795100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce4005040 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cfbe00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2cec002a80 tx=0x7f2cec004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec007cc0 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cec00f040 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc119a50 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.291+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec00dad0 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.292+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc119fc0 con 0x7f2cfc072820 2024-04-16T07:16:28.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.293+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc04c710 con 0x7f2cfc072820 2024-04-16T07:16:28.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.294+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2cec00dc30 con 0x7f2cfc072820 2024-04-16T07:16:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.294+0000 7f2cf8c00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 0x7f2ce006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:16:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.295+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f2cec086730 con 0x7f2cfc072820 2024-04-16T07:16:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.295+0000 7f2cf3e00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 0x7f2ce006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:16:28.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.296+0000 7f2cf3e00700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 0x7f2ce006b750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4004f90 tx=0x7f2ce4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:16:28.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.300+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2cec056c90 con 0x7f2cfc072820 2024-04-16T07:16:28.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:28.517+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cfc10a480 con 0x7f2ce00692a0 2024-04-16T07:16:29.006 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:28 smithi026 sudo[63877]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T07:16:29.007 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:28 smithi026 sudo[63877]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:29.007 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:28 smithi026 sudo[63877]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:29.007 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:28 smithi026 sudo[63877]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:29.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 ceph-mon[28805]: mgrmap e19: smithi026.tawexg(active, since 2m), standbys: smithi104.uqfljg 2024-04-16T07:16:29.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 ceph-mon[28805]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:16:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:16:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:16:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:29 smithi104 ceph-mon[36163]: mgrmap e19: smithi026.tawexg(active, since 2m), standbys: smithi104.uqfljg 2024-04-16T07:16:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:29 smithi104 ceph-mon[36163]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:16:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:29 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:16:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:29 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:16:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:29 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:29.760 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66286]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T07:16:29.760 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66286]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:29.760 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66286]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:29.760 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66286]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:30.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T07:16:30.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:30.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66294]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:30.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:29 smithi026 sudo[66294]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:30 smithi026 sudo[66297]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T07:16:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:30 smithi026 sudo[66297]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:31.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:30 smithi026 sudo[66297]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:31.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:30 smithi026 sudo[66297]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T07:16:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39442]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39442]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: pgmap v100: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi026 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:16:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:16:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39468]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T07:16:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39468]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 sudo[39468]: pam_unix(sudo:session): session closed for user root 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: pgmap v100: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi026 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:16:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 99 KiB/s wr, 6 op/s 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi026 to 4290M 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/87743777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]: dispatch 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]': finished 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T07:16:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:33.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T07:16:33.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 99 KiB/s wr, 6 op/s 2024-04-16T07:16:33.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi026 to 4290M 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/87743777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]: dispatch 2024-04-16T07:16:33.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]: dispatch 2024-04-16T07:16:33.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e51da929-01c2-498d-9c12-e4bb3ea1e4d7"}]': finished 2024-04-16T07:16:33.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T07:16:33.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:34.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:33 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3110690232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:16:34.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:33 smithi026 ceph-mon[28805]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2024-04-16T07:16:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:33 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/3110690232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:16:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:33 smithi104 ceph-mon[36163]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2024-04-16T07:16:36.343 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:36 smithi026 ceph-mon[28805]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 994 B/s rd, 56 KiB/s wr, 3 op/s 2024-04-16T07:16:36.572 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:36 smithi104 ceph-mon[36163]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 994 B/s rd, 56 KiB/s wr, 3 op/s 2024-04-16T07:16:38.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T07:16:38.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:38.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:38.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:38.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:38.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:38.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T07:16:38.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:38.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:38.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:38.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:38.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:40.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:40 smithi104 ceph-mon[36163]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T07:16:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:40 smithi026 ceph-mon[28805]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T07:16:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:41 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:16:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:41 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:41 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:16:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:41 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:42.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:42 smithi104 ceph-mon[36163]: Deploying daemon osd.3 on smithi104 2024-04-16T07:16:42.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:42 smithi104 ceph-mon[36163]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T07:16:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:42 smithi026 ceph-mon[28805]: Deploying daemon osd.3 on smithi104 2024-04-16T07:16:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:42 smithi026 ceph-mon[28805]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T07:16:44.744 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:44 smithi104 ceph-mon[36163]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-04-16T07:16:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:44 smithi026 ceph-mon[28805]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-04-16T07:16:46.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:46 smithi026 ceph-mon[28805]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T07:16:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:46 smithi104 ceph-mon[36163]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T07:16:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:16:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:48 smithi104 ceph-mon[36163]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T07:16:48.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:48 smithi026 ceph-mon[28805]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T07:16:50.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:50.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:50.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:50.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:49 smithi026 ceph-mon[28805]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:16:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:49 smithi104 ceph-mon[36163]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:52.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:52 smithi104 ceph-mon[36163]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:52.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:52 smithi026 ceph-mon[28805]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:53.475 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:53 smithi104 ceph-mon[36163]: from='osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:16:53.475 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:53 smithi104 ceph-mon[36163]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:16:53.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.568+0000 7f2cf8c00700 1 -- 172.21.15.104:0/321795100 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2cfc10a480 con 0x7f2ce00692a0 2024-04-16T07:16:53.567 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 3 on host 'smithi104' 2024-04-16T07:16:53.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 msgr2=0x7f2ce006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:53.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 0x7f2ce006b750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4004f90 tx=0x7f2ce4009e30 comp rx=0 tx=0).stop 2024-04-16T07:16:53.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 msgr2=0x7f2cfc113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2cec002a80 tx=0x7f2cec004600 comp rx=0 tx=0).stop 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 shutdown_connections 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ce00692a0 0x7f2ce006b750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc072820 0x7f2cfc113f40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 --2- 172.21.15.104:0/321795100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cfc114480 0x7f2cfc1194b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.571+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 >> 172.21.15.104:0/321795100 conn(0x7f2cfc0fa950 msgr2=0x7f2cfc0fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.572+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 shutdown_connections 2024-04-16T07:16:53.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:16:53.572+0000 7f2d03200700 1 -- 172.21.15.104:0/321795100 wait complete. 2024-04-16T07:16:53.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:53 smithi026 ceph-mon[28805]: from='osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:16:53.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:53 smithi026 ceph-mon[28805]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:16:54.318 DEBUG:teuthology.orchestra.run.smithi104:osd.3> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.3.service 2024-04-16T07:16:54.324 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_3... 2024-04-16T07:16:54.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:54 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.663 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:54.663 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T07:16:54.663 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:54.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:54 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:55.699 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:55 smithi026 ceph-mon[28805]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:16:55.699 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:55 smithi026 ceph-mon[28805]: osdmap e26: 4 total, 3 up, 4 in 2024-04-16T07:16:55.699 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:55 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:55.699 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:55 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:55.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:55 smithi104 ceph-mon[36163]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:16:55.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:55 smithi104 ceph-mon[36163]: osdmap e26: 4 total, 3 up, 4 in 2024-04-16T07:16:55.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:55 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:55.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:55 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:55.776 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] boot 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:56 smithi026 ceph-mon[28805]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: osd.3 [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] boot 2024-04-16T07:16:56.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T07:16:56.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:16:56.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:56 smithi104 ceph-mon[36163]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T07:16:57.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:57 smithi104 ceph-mon[36163]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T07:16:57.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:57 smithi026 ceph-mon[28805]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T07:16:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:58 smithi104 ceph-mon[36163]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:16:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:58 smithi104 ceph-mon[36163]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T07:16:58.914 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T07:16:58.914 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T07:16:58.914 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-16T07:16:58.915 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-16T07:16:58.915 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0363504 s, 288 MB/s 2024-04-16T07:16:58.915 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-16T07:16:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:58 smithi026 ceph-mon[28805]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:16:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:58 smithi026 ceph-mon[28805]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T07:16:59.752 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: osdmap e31: 4 total, 4 up, 4 in 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi104 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi104 to 15937M 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:16:59.753 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:16:59 smithi104 ceph-mon[36163]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:16:59.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-04-16T07:17:00.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: osdmap e31: 4 total, 4 up, 4 in 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi104 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi104 to 15937M 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:00.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:16:59 smithi026 ceph-mon[28805]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:00.245 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:17:00.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:00.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:00.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:00.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:00 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:00 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:01.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:01 smithi104 ceph-mon[36163]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-16T07:17:01.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:01 smithi026 ceph-mon[28805]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-16T07:17:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:02.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 -- 172.21.15.104:0/2768840762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62880708f0 msgr2=0x7f6288070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:02.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 --2- 172.21.15.104:0/2768840762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62880708f0 0x7f6288070d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6274004970 tx=0x7f6274009e30 comp rx=0 tx=0).stop 2024-04-16T07:17:02.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 -- 172.21.15.104:0/2768840762 shutdown_connections 2024-04-16T07:17:02.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 --2- 172.21.15.104:0/2768840762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62880708f0 0x7f6288070d60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 --2- 172.21.15.104:0/2768840762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6288071ea0 0x7f62880703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 -- 172.21.15.104:0/2768840762 >> 172.21.15.104:0/2768840762 conn(0x7f62880f8580 msgr2=0x7f62880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 -- 172.21.15.104:0/2768840762 shutdown_connections 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.354+0000 7f628ce00700 1 -- 172.21.15.104:0/2768840762 wait complete. 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 Processor -- start 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 -- start start 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62880708f0 0x7f628810fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6288110640 con 0x7f62880708f0 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.355+0000 7f628ce00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62881107b0 con 0x7f6288071ea0 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33866/0 (socket says 172.21.15.104:33866) 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 -- 172.21.15.104:0/313971150 learned_addr learned my addr 172.21.15.104:0/313971150 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285e00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62880708f0 0x7f628810fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:02.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 -- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62880708f0 msgr2=0x7f628810fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62880708f0 0x7f628810fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 -- 172.21.15.104:0/313971150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274005040 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.356+0000 7f6285400700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6274004f50 tx=0x7f6274009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.357+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627400e040 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.357+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f627400bce0 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.357+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627401c630 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.357+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62881151b0 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.357+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6288115770 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.358+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628804c710 con 0x7f6288071ea0 2024-04-16T07:17:02.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.359+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6274013a80 con 0x7f6288071ea0 2024-04-16T07:17:02.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.360+0000 7f627e000700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 0x7f626c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:02.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.360+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f6274087e90 con 0x7f6288071ea0 2024-04-16T07:17:02.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.360+0000 7f6285e00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 0x7f626c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:02.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.362+0000 7f6285e00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 0x7f626c06b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6288071be0 tx=0x7f627800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:17:02.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.365+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f62740582d0 con 0x7f6288071ea0 2024-04-16T07:17:02.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:02.586+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6288103d20 con 0x7f626c0692a0 2024-04-16T07:17:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:17:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:17:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:03.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:17:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:17:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:04.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:03 smithi026 ceph-mon[28805]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:17:04.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:03 smithi026 ceph-mon[28805]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-16T07:17:04.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:03 smithi104 ceph-mon[36163]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:17:04.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:03 smithi104 ceph-mon[36163]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/3294012973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]: dispatch 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]: dispatch 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]': finished 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T07:17:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:06 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:06.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T07:17:06.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/3294012973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]: dispatch 2024-04-16T07:17:06.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]: dispatch 2024-04-16T07:17:06.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d46a520-bde2-4ef9-9420-1ebb0edbe1ea"}]': finished 2024-04-16T07:17:06.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T07:17:06.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:06 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:07.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:07 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/435802184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:17:07.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:07 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/435802184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:17:08.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:08 smithi026 ceph-mon[28805]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-16T07:17:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:08 smithi104 ceph-mon[36163]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-16T07:17:10.715 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:10 smithi104 ceph-mon[36163]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-16T07:17:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:10 smithi026 ceph-mon[28805]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-16T07:17:12.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:12 smithi104 ceph-mon[36163]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:12.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:12 smithi026 ceph-mon[28805]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:14.657 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:14 smithi104 ceph-mon[36163]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:14 smithi026 ceph-mon[28805]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:15.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:17:15.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:15.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:17:15.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:16 smithi026 ceph-mon[28805]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:16 smithi026 ceph-mon[28805]: Deploying daemon osd.4 on smithi104 2024-04-16T07:17:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:16 smithi104 ceph-mon[36163]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:16 smithi104 ceph-mon[36163]: Deploying daemon osd.4 on smithi104 2024-04-16T07:17:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:18.754 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:18 smithi104 ceph-mon[36163]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:18 smithi026 ceph-mon[28805]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:20.711 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:20 smithi104 ceph-mon[36163]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:20 smithi026 ceph-mon[28805]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:22 smithi104 ceph-mon[36163]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:22 smithi026 ceph-mon[28805]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:24.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:24.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:24.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:24.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:24 smithi104 ceph-mon[36163]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:24 smithi026 ceph-mon[28805]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:26.555 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:26 smithi104 ceph-mon[36163]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:26 smithi026 ceph-mon[28805]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.341+0000 7f627e000700 1 -- 172.21.15.104:0/313971150 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6288103d20 con 0x7f626c0692a0 2024-04-16T07:17:27.340 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-04-16T07:17:27.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 msgr2=0x7f626c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 0x7f626c06b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6288071be0 tx=0x7f627800b040 comp rx=0 tx=0).stop 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 msgr2=0x7f6288110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6274004f50 tx=0x7f6274009e30 comp rx=0 tx=0).stop 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 shutdown_connections 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f626c0692a0 0x7f626c06b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62880708f0 0x7f628810fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 --2- 172.21.15.104:0/313971150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6288071ea0 0x7f6288110040 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 >> 172.21.15.104:0/313971150 conn(0x7f62880f8580 msgr2=0x7f62880f9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 shutdown_connections 2024-04-16T07:17:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:27.344+0000 7f628ce00700 1 -- 172.21.15.104:0/313971150 wait complete. 2024-04-16T07:17:27.345 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:27 smithi104 ceph-mon[36163]: from='osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:17:27.345 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:27 smithi104 ceph-mon[36163]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:17:27.345 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:27 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:27.345 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:27 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:27 smithi026 ceph-mon[28805]: from='osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:17:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:27 smithi026 ceph-mon[28805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:17:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:27 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:27 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:28.062 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.4.service 2024-04-16T07:17:28.068 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_2... 2024-04-16T07:17:28.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T07:17:28.535 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:28.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:28.680 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[48085]: 2024-04-16T07:17:28.370+0000 7f3a10a00700 -1 osd.4 0 waiting for initial osdmap 2024-04-16T07:17:28.680 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:17:28 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[48085]: 2024-04-16T07:17:28.383+0000 7f3a07e00700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:17:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T07:17:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T07:17:28.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:28.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:17:28.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:17:28.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:28.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:28 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:29.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:29 smithi104 ceph-mon[36163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:17:29.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:29 smithi104 ceph-mon[36163]: osdmap e34: 5 total, 4 up, 5 in 2024-04-16T07:17:29.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:29 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:29.749 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:29 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:29 smithi026 ceph-mon[28805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:17:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:29 smithi026 ceph-mon[28805]: osdmap e34: 5 total, 4 up, 5 in 2024-04-16T07:17:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:29 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:29 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] boot 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T07:17:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:30 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:17:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:17:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T07:17:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: osd.4 [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] boot 2024-04-16T07:17:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T07:17:30.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:30 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:17:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T07:17:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T07:17:31.509 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-16T07:17:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-16T07:17:31.510 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0341381 s, 307 MB/s 2024-04-16T07:17:31.510 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-16T07:17:31.510 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:31 smithi104 ceph-mon[36163]: osdmap e36: 5 total, 5 up, 5 in 2024-04-16T07:17:31.510 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:31 smithi026 ceph-mon[28805]: osdmap e36: 5 total, 5 up, 5 in 2024-04-16T07:17:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:32.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:32 smithi104 ceph-mon[36163]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:32.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-04-16T07:17:32.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:32 smithi026 ceph-mon[28805]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:33.179 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:17:34.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:34 smithi104 ceph-mon[36163]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:34 smithi026 ceph-mon[28805]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 -- 172.21.15.104:0/2584184722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad0100070 msgr2=0x7f6ad01004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/2584184722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad0100070 0x7f6ad01004e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6abc004940 tx=0x7f6abc009e30 comp rx=0 tx=0).stop 2024-04-16T07:17:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 -- 172.21.15.104:0/2584184722 shutdown_connections 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/2584184722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad0100070 0x7f6ad01004e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/2584184722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad00fef00 0x7f6ad00ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 -- 172.21.15.104:0/2584184722 >> 172.21.15.104:0/2584184722 conn(0x7f6ad00fa790 msgr2=0x7f6ad00fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 -- 172.21.15.104:0/2584184722 shutdown_connections 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.868+0000 7f6ad5e00700 1 -- 172.21.15.104:0/2584184722 wait complete. 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.869+0000 7f6ad5e00700 1 Processor -- start 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.869+0000 7f6ad5e00700 1 -- start start 2024-04-16T07:17:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 0x7f6ad010ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad5e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01105e0 con 0x7f6ad0100070 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad5e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0110e30 con 0x7f6ad00fef00 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50838/0 (socket says 172.21.15.104:50838) 2024-04-16T07:17:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 -- 172.21.15.104:0/454142132 learned_addr learned my addr 172.21.15.104:0/454142132 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:17:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6acfe00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 0x7f6ad010ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 -- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 msgr2=0x7f6ad010ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:17:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 0x7f6ad010ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6ad4a00700 1 -- 172.21.15.104:0/454142132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc005040 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.870+0000 7f6acfe00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 0x7f6ad010ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6ad4a00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4002a50 tx=0x7f6ac40046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac4007cc0 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad0111110 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ac400f040 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad0111650 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.871+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac4009600 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.872+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad00fe290 con 0x7f6ad00fef00 2024-04-16T07:17:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.873+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6ac4009840 con 0x7f6ad00fef00 2024-04-16T07:17:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.874+0000 7f6accc00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 0x7f6ab806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:17:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.874+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f6ac4086a80 con 0x7f6ad00fef00 2024-04-16T07:17:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.875+0000 7f6acfe00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 0x7f6ab806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:17:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.876+0000 7f6acfe00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 0x7f6ab806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00b7a0 tx=0x7f6abc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:17:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:35.879+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6ac4056cb0 con 0x7f6ad00fef00 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi104 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi104 to 7968M 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:36.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:35 smithi026 ceph-mon[28805]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi104 2024-04-16T07:17:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi104 to 7968M 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:35 smithi104 ceph-mon[36163]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:17:36.100+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ad01119d0 con 0x7f6ab80692a0 2024-04-16T07:17:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:36 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:37.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:17:37.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T07:17:37.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T07:17:37.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:37.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:37.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:17:37.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:36 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:37.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:37 smithi104 ceph-mon[36163]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:38.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:37 smithi026 ceph-mon[28805]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/820269060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]: dispatch 2024-04-16T07:17:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]: dispatch 2024-04-16T07:17:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]': finished 2024-04-16T07:17:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T07:17:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:40 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:17:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/820269060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]: dispatch 2024-04-16T07:17:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]: dispatch 2024-04-16T07:17:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f752a2e-4458-4cde-821d-59a8d703f253"}]': finished 2024-04-16T07:17:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T07:17:40.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:40 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:17:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:41 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/4030933665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:17:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:41 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/4030933665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T07:17:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:42 smithi026 ceph-mon[28805]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:42 smithi104 ceph-mon[36163]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:44.635 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:44 smithi104 ceph-mon[36163]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:44 smithi026 ceph-mon[28805]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:46.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:46 smithi026 ceph-mon[28805]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:46 smithi104 ceph-mon[36163]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:17:48.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:48 smithi026 ceph-mon[28805]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:48.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:48 smithi104 ceph-mon[36163]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:49.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:17:49.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:49 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:49.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:17:49.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:49 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:17:50.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:50 smithi026 ceph-mon[28805]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:50.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:50 smithi026 ceph-mon[28805]: Deploying daemon osd.5 on smithi104 2024-04-16T07:17:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:50 smithi104 ceph-mon[36163]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:50 smithi104 ceph-mon[36163]: Deploying daemon osd.5 on smithi104 2024-04-16T07:17:52.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:52 smithi026 ceph-mon[28805]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:52.820 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:52 smithi104 ceph-mon[36163]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:54.517 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:54 smithi104 ceph-mon[36163]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:54.562 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:54 smithi026 ceph-mon[28805]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:56.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:56 smithi026 ceph-mon[28805]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:56.770 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:56 smithi104 ceph-mon[36163]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:58 smithi104 ceph-mon[36163]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:17:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:58 smithi026 ceph-mon[28805]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:17:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:17:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:17:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:17:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:00.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:00 smithi104 ceph-mon[36163]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:00.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:00 smithi026 ceph-mon[28805]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:01.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:01 smithi104 ceph-mon[36163]: from='osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:18:01.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:01 smithi104 ceph-mon[36163]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:18:01.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:01 smithi026 ceph-mon[28805]: from='osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:18:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:01 smithi026 ceph-mon[28805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:18:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:02.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.092+0000 7f6accc00700 1 -- 172.21.15.104:0/454142132 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6ad01119d0 con 0x7f6ab80692a0 2024-04-16T07:18:02.091 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 msgr2=0x7f6ab806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 0x7f6ab806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00b7a0 tx=0x7f6abc009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 msgr2=0x7f6ad010faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4002a50 tx=0x7f6ac40046b0 comp rx=0 tx=0).stop 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 shutdown_connections 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ab80692a0 0x7f6ab806b750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad00fef00 0x7f6ad010faa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 --2- 172.21.15.104:0/454142132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ad0100070 0x7f6ad010ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.095+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 >> 172.21.15.104:0/454142132 conn(0x7f6ad00fa790 msgr2=0x7f6ad00fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.096+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 shutdown_connections 2024-04-16T07:18:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:02.096+0000 7f6ad5e00700 1 -- 172.21.15.104:0/454142132 wait complete. 2024-04-16T07:18:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:02 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T07:18:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T07:18:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:18:02.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:18:02.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:02.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:02 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:02.912 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.5.service 2024-04-16T07:18:02.916 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-16T07:18:02.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-16T07:18:03.429 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: osdmap e39: 6 total, 5 up, 6 in 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:03.513 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:03 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: osdmap e39: 6 total, 5 up, 6 in 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:03.607 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:03 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: purged_snaps scrub starts 2024-04-16T07:18:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: purged_snaps scrub ok 2024-04-16T07:18:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] boot 2024-04-16T07:18:04.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T07:18:04.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:04 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:04.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: purged_snaps scrub starts 2024-04-16T07:18:04.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: purged_snaps scrub ok 2024-04-16T07:18:04.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T07:18:04.618 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: osd.5 [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] boot 2024-04-16T07:18:04.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T07:18:04.619 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:04 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.635+0000 7f62e9400700 1 -- 172.21.15.26:0/2878940708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fdff0 msgr2=0x7f62e40fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.635+0000 7f62e9400700 1 --2- 172.21.15.26:0/2878940708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fdff0 0x7f62e40fe460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62d0004940 tx=0x7f62d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 -- 172.21.15.26:0/2878940708 shutdown_connections 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 --2- 172.21.15.26:0/2878940708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fdff0 0x7f62e40fe460 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 --2- 172.21.15.26:0/2878940708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fce80 0x7f62e40fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 -- 172.21.15.26:0/2878940708 >> 172.21.15.26:0/2878940708 conn(0x7f62e40f8580 msgr2=0x7f62e40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 -- 172.21.15.26:0/2878940708 shutdown_connections 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 -- 172.21.15.26:0/2878940708 wait complete. 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.636+0000 7f62e9400700 1 Processor -- start 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e9400700 1 -- start start 2024-04-16T07:18:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e9400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fdff0 0x7f62e4110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e9400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4110680 con 0x7f62e40fce80 2024-04-16T07:18:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e9400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e41107f0 con 0x7f62e40fdff0 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e2800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e2800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38182/0 (socket says 172.21.15.26:38182) 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.637+0000 7f62e2800700 1 -- 172.21.15.26:0/4159844472 learned_addr learned my addr 172.21.15.26:0/4159844472 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.638+0000 7f62e1e00700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fdff0 0x7f62e4110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.638+0000 7f62e2800700 1 -- 172.21.15.26:0/4159844472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fdff0 msgr2=0x7f62e4110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.638+0000 7f62e2800700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fdff0 0x7f62e4110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.638+0000 7f62e2800700 1 -- 172.21.15.26:0/4159844472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d0005040 con 0x7f62e40fce80 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.638+0000 7f62e2800700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62d4002a50 tx=0x7f62d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.639+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d4007cc0 con 0x7f62e40fce80 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.639+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62d400f040 con 0x7f62e40fce80 2024-04-16T07:18:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.639+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e4115250 con 0x7f62e40fce80 2024-04-16T07:18:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.639+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d400da30 con 0x7f62e40fce80 2024-04-16T07:18:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.639+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e4115810 con 0x7f62e40fce80 2024-04-16T07:18:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.640+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4101980 con 0x7f62e40fce80 2024-04-16T07:18:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.641+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f62d400db90 con 0x7f62e40fce80 2024-04-16T07:18:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.642+0000 7f62daa00700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 0x7f62c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.642+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f62d4086bd0 con 0x7f62e40fce80 2024-04-16T07:18:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.642+0000 7f62e1e00700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 0x7f62c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.643+0000 7f62e1e00700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 0x7f62c806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62d000b3b0 tx=0x7f62d0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.647+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f62d4056c30 con 0x7f62e40fce80 2024-04-16T07:18:05.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:05 smithi026 ceph-mon[28805]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T07:18:05.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:05 smithi104 ceph-mon[36163]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T07:18:05.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.866+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f62e404c710 con 0x7f62e40fce80 2024-04-16T07:18:05.866 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:18:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.867+0000 7f62daa00700 1 -- 172.21.15.26:0/4159844472 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f62d40567c0 con 0x7f62e40fce80 2024-04-16T07:18:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 msgr2=0x7f62c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 0x7f62c806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62d000b3b0 tx=0x7f62d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 msgr2=0x7f62e410fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62d4002a50 tx=0x7f62d40046b0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 shutdown_connections 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f62c80692a0 0x7f62c806b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62e40fce80 0x7f62e410fb40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 --2- 172.21.15.26:0/4159844472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62e40fdff0 0x7f62e4110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 >> 172.21.15.26:0/4159844472 conn(0x7f62e40f8580 msgr2=0x7f62e40f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.870+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 shutdown_connections 2024-04-16T07:18:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:05.871+0000 7f62e9400700 1 -- 172.21.15.26:0/4159844472 wait complete. 2024-04-16T07:18:06.669 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1713251883,"num_in_osds":6,"osd_in_since":1713251859,"num_remapped_pgs":0} 2024-04-16T07:18:06.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T07:18:06.703 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:06 smithi104 ceph-mon[36163]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:06.703 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:06 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/4159844472' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T07:18:06.752 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:06 smithi026 ceph-mon[28805]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:06.752 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:06 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4159844472' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T07:18:07.160 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:08.727 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:18:08.728 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:18:08.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.326+0000 7f8af4600700 1 -- 172.21.15.26:0/3650599239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 msgr2=0x7f8aec103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.326+0000 7f8af4600700 1 --2- 172.21.15.26:0/3650599239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec103230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0004940 tx=0x7f8ae0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 -- 172.21.15.26:0/3650599239 shutdown_connections 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 --2- 172.21.15.26:0/3650599239 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 --2- 172.21.15.26:0/3650599239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec103230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 -- 172.21.15.26:0/3650599239 >> 172.21.15.26:0/3650599239 conn(0x7f8aec072bc0 msgr2=0x7f8aec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:09.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 -- 172.21.15.26:0/3650599239 shutdown_connections 2024-04-16T07:18:09.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.327+0000 7f8af4600700 1 -- 172.21.15.26:0/3650599239 wait complete. 2024-04-16T07:18:09.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.328+0000 7f8af4600700 1 Processor -- start 2024-04-16T07:18:09.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.328+0000 7f8af4600700 1 -- start start 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af4600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af4600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af4600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec1168f0 con 0x7f8aec102e60 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af4600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec10fcc0 con 0x7f8aec0656b0 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47644/0 (socket says 172.21.15.26:47644) 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1200700 1 -- 172.21.15.26:0/1456697146 learned_addr learned my addr 172.21.15.26:0/1456697146 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1c00700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1200700 1 -- 172.21.15.26:0/1456697146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 msgr2=0x7f8aec115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.329+0000 7f8af1200700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8af1200700 1 -- 172.21.15.26:0/1456697146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0005040 con 0x7f8aec102e60 2024-04-16T07:18:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8af1c00700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8af1200700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8002a50 tx=0x7f8ad80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad8007cc0 con 0x7f8aec102e60 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ad800f040 con 0x7f8aec102e60 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad8009640 con 0x7f8aec102e60 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec10ffa0 con 0x7f8aec102e60 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.330+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec110560 con 0x7f8aec102e60 2024-04-16T07:18:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.331+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec107320 con 0x7f8aec102e60 2024-04-16T07:18:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.333+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8ad80097a0 con 0x7f8aec102e60 2024-04-16T07:18:09.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.334+0000 7f8ae9600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 0x7f8ad406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:09.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.334+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8ad8086c20 con 0x7f8aec102e60 2024-04-16T07:18:09.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.334+0000 7f8af1c00700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 0x7f8ad406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:09.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.335+0000 7f8af1c00700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 0x7f8ad406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0004560 tx=0x7f8ae0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:09.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.338+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8ad8056c80 con 0x7f8aec102e60 2024-04-16T07:18:09.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.559+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aec0fe570 con 0x7f8aec102e60 2024-04-16T07:18:09.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.561+0000 7f8ae9600700 1 -- 172.21.15.26:0/1456697146 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11085 (secure 0 0 0) 0x7f8ad8056810 con 0x7f8aec102e60 2024-04-16T07:18:09.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:18:09.561 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:18:04.421019+0000","last_up_change":"2024-04-16T07:18:03.334452+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6803","nonce":3225147345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6805","nonce":3225147345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6809","nonce":3225147345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6807","nonce":3225147345}]},"public_addr":"172.21.15.26:6803/3225147345","cluster_addr":"172.21.15.26:6805/3225147345","heartbeat_back_addr":"172.21.15.26:6809/3225147345","heartbeat_front_addr":"172.21.15.26:6807/3225147345","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6811","nonce":1194632492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6813","nonce":1194632492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6817","nonce":1194632492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6815","nonce":1194632492}]},"public_addr":"172.21.15.26:6811/1194632492","cluster_addr":"172.21.15.26:6813/1194632492","heartbeat_back_addr":"172.21.15.26:6817/1194632492","heartbeat_front_addr":"172.21.15.26:6815/1194632492","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6819","nonce":828513924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6821","nonce":828513924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6825","nonce":828513924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6823","nonce":828513924}]},"public_addr":"172.21.15.26:6819/828513924","cluster_addr":"172.21.15.26:6821/828513924","heartbeat_back_addr":"172.21.15.26:6825/828513924","heartbeat_front_addr":"172.21.15.26:6823/828513924","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6801","nonce":2432746791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6803","nonce":2432746791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6807","nonce":2432746791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6805","nonce":2432746791}]},"public_addr":"172.21.15.104:6801/2432746791","cluster_addr":"172.21.15.104:6803/2432746791","heartbeat_back_addr":"172.21.15.104:6807/2432746791","heartbeat_front_addr":"172.21.15.104:6805/2432746791","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6809","nonce":3400087284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6811","nonce":3400087284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6815","nonce":3400087284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6813","nonce":3400087284}]},"public_addr":"172.21.15.104:6809/3400087284","cluster_addr":"172.21.15.104:6811/3400087284","heartbeat_back_addr":"172.21.15.104:6815/3400087284","heartbeat_front_addr":"172.21.15.104:6813/3400087284","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6817","nonce":4159292069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6819","nonce":4159292069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6823","nonce":4159292069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6821","nonce":4159292069}]},"public_addr":"172.21.15.104:6817/4159292069","cluster_addr":"172.21.15.104:6819/4159292069","heartbeat_back_addr":"172.21.15.104:6823/4159292069","heartbeat_front_addr":"172.21.15.104:6821/4159292069","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 msgr2=0x7f8ad406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 0x7f8ad406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0004560 tx=0x7f8ae0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 msgr2=0x7f8aec1162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8002a50 tx=0x7f8ad80046b0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 shutdown_connections 2024-04-16T07:18:09.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8ad40692a0 0x7f8ad406b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8aec0656b0 0x7f8aec115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 --2- 172.21.15.26:0/1456697146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8aec102e60 0x7f8aec1162a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:09.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 >> 172.21.15.26:0/1456697146 conn(0x7f8aec072bc0 msgr2=0x7f8aec0fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:09.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 shutdown_connections 2024-04-16T07:18:09.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:09.565+0000 7f8af4600700 1 -- 172.21.15.26:0/1456697146 wait complete. 2024-04-16T07:18:09.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:09 smithi026 ceph-mon[28805]: Detected new or changed devices on smithi104 2024-04-16T07:18:09.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:09 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi104 to 5312M 2024-04-16T07:18:09.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:09 smithi104 ceph-mon[36163]: Detected new or changed devices on smithi104 2024-04-16T07:18:09.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:09 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi104 to 5312M 2024-04-16T07:18:10.299 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T07:16:23.932743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-16T07:18:10.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-16T07:18:10.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:10 smithi026 ceph-mon[28805]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:10.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:10 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1456697146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:10 smithi104 ceph-mon[36163]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:10 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1456697146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:12.595 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:12 smithi026 ceph-mon[28805]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:12 smithi104 ceph-mon[36163]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 -- 172.21.15.26:0/4140989947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc102e20 msgr2=0x7f22bc1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 --2- 172.21.15.26:0/4140989947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc102e20 0x7f22bc1031f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22ac004940 tx=0x7f22ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 -- 172.21.15.26:0/4140989947 shutdown_connections 2024-04-16T07:18:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 --2- 172.21.15.26:0/4140989947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc1037c0 0x7f22bc0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 --2- 172.21.15.26:0/4140989947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc102e20 0x7f22bc1031f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 -- 172.21.15.26:0/4140989947 >> 172.21.15.26:0/4140989947 conn(0x7f22bc0f89d0 msgr2=0x7f22bc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 -- 172.21.15.26:0/4140989947 shutdown_connections 2024-04-16T07:18:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.083+0000 7f22c2e00700 1 -- 172.21.15.26:0/4140989947 wait complete. 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.084+0000 7f22c2e00700 1 Processor -- start 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.084+0000 7f22c2e00700 1 -- start start 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22c2e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 0x7f22bc115dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22c2e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22c2e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc116950 con 0x7f22bc1037c0 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22c2e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc10fd20 con 0x7f22bc102e20 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22b3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22b3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47672/0 (socket says 172.21.15.26:47672) 2024-04-16T07:18:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22b3400700 1 -- 172.21.15.26:0/760886586 learned_addr learned my addr 172.21.15.26:0/760886586 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.085+0000 7f22bbe00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 0x7f22bc115dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22b3400700 1 -- 172.21.15.26:0/760886586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 msgr2=0x7f22bc115dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22b3400700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 0x7f22bc115dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22b3400700 1 -- 172.21.15.26:0/760886586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac005040 con 0x7f22bc1037c0 2024-04-16T07:18:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22bbe00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 0x7f22bc115dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22b3400700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22a4002a50 tx=0x7f22a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.086+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22a4007cc0 con 0x7f22bc1037c0 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.087+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22a400f040 con 0x7f22bc1037c0 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.087+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22a400daf0 con 0x7f22bc1037c0 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.087+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc110000 con 0x7f22bc1037c0 2024-04-16T07:18:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.087+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc110570 con 0x7f22bc1037c0 2024-04-16T07:18:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.088+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc106a20 con 0x7f22bc1037c0 2024-04-16T07:18:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.089+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f22a4009810 con 0x7f22bc1037c0 2024-04-16T07:18:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.090+0000 7f22b8c00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 0x7f22a006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.090+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f22a40869e0 con 0x7f22bc1037c0 2024-04-16T07:18:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.090+0000 7f22bbe00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 0x7f22a006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.091+0000 7f22bbe00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 0x7f22a006b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22ac000c00 tx=0x7f22ac009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.095+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f22a4056a60 con 0x7f22bc1037c0 2024-04-16T07:18:13.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.309+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22bc111470 con 0x7f22bc1037c0 2024-04-16T07:18:13.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.310+0000 7f22b8c00700 1 -- 172.21.15.26:0/760886586 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22a40565f0 con 0x7f22bc1037c0 2024-04-16T07:18:13.309 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 msgr2=0x7f22a006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 0x7f22a006b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22ac000c00 tx=0x7f22ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 msgr2=0x7f22bc116300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22a4002a50 tx=0x7f22a4004600 comp rx=0 tx=0).stop 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 shutdown_connections 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f22a00692b0 0x7f22a006b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22bc102e20 0x7f22bc115dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 --2- 172.21.15.26:0/760886586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22bc1037c0 0x7f22bc116300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:13.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.313+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 >> 172.21.15.26:0/760886586 conn(0x7f22bc0f89d0 msgr2=0x7f22bc107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:13.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.314+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 shutdown_connections 2024-04-16T07:18:13.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:13.314+0000 7f22c2e00700 1 -- 172.21.15.26:0/760886586 wait complete. 2024-04-16T07:18:13.635 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:13 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/760886586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:18:13.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:13 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/760886586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:18:14.052 INFO:tasks.cephadm:Setting up client nodes... 2024-04-16T07:18:14.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T07:18:14.570 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:14.721 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:14 smithi026 ceph-mon[28805]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:14.733 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:14 smithi104 ceph-mon[36163]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:16.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:16 smithi104 ceph-mon[36163]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:16.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 -- 172.21.15.26:0/2435016704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 msgr2=0x7f9efc103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 --2- 172.21.15.26:0/2435016704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc103830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9eec004940 tx=0x7f9eec009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 -- 172.21.15.26:0/2435016704 shutdown_connections 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 --2- 172.21.15.26:0/2435016704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 --2- 172.21.15.26:0/2435016704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc103830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 -- 172.21.15.26:0/2435016704 >> 172.21.15.26:0/2435016704 conn(0x7f9efc072640 msgr2=0x7f9efc074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:16.729 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:16 smithi026 ceph-mon[28805]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:16.729 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.729+0000 7f9f02200700 1 -- 172.21.15.26:0/2435016704 shutdown_connections 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.730+0000 7f9f02200700 1 -- 172.21.15.26:0/2435016704 wait complete. 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.730+0000 7f9f02200700 1 Processor -- start 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9f02200700 1 -- start start 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9f02200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9f02200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9f02200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc114aa0 con 0x7f9efc103420 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9f02200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc114c10 con 0x7f9efc1046d0 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9efb400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9efb400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47692/0 (socket says 172.21.15.26:47692) 2024-04-16T07:18:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.731+0000 7f9efb400700 1 -- 172.21.15.26:0/528200157 learned_addr learned my addr 172.21.15.26:0/528200157 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efaa00700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efb400700 1 -- 172.21.15.26:0/528200157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 msgr2=0x7f9efc1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efb400700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc1144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efb400700 1 -- 172.21.15.26:0/528200157 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec005040 con 0x7f9efc103420 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efaa00700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.732+0000 7f9efb400700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009150 tx=0x7f9eec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.733+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eec00e040 con 0x7f9efc103420 2024-04-16T07:18:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.733+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9eec00bbf0 con 0x7f9efc103420 2024-04-16T07:18:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.733+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eec01c630 con 0x7f9efc103420 2024-04-16T07:18:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.733+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc119610 con 0x7f9efc103420 2024-04-16T07:18:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.733+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc119bd0 con 0x7f9efc103420 2024-04-16T07:18:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.735+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc04c710 con 0x7f9efc103420 2024-04-16T07:18:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.736+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9eec00bd60 con 0x7f9efc103420 2024-04-16T07:18:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.737+0000 7f9ef3400700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 0x7f9ee006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.737+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9eec088250 con 0x7f9efc103420 2024-04-16T07:18:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.737+0000 7f9efaa00700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 0x7f9ee006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.738+0000 7f9efaa00700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 0x7f9ee006b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8007fd0 tx=0x7f9ee8009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:16.742+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9eec058330 con 0x7f9efc103420 2024-04-16T07:18:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.069+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9efc10bf10 con 0x7f9efc103420 2024-04-16T07:18:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.077+0000 7f9ef3400700 1 -- 172.21.15.26:0/528200157 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9eec054020 con 0x7f9efc103420 2024-04-16T07:18:17.076 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-04-16T07:18:17.076 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQA5Jh5mpDhDBBAATwv2HLN4Sv0PpG7MnxpdMg== 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 msgr2=0x7f9ee006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 0x7f9ee006b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8007fd0 tx=0x7f9ee8009040 comp rx=0 tx=0).stop 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 msgr2=0x7f9efc113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009150 tx=0x7f9eec009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 shutdown_connections 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9ee00692b0 0x7f9ee006b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9efc103420 0x7f9efc113f60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:17.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 --2- 172.21.15.26:0/528200157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9efc1046d0 0x7f9efc1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:17.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 >> 172.21.15.26:0/528200157 conn(0x7f9efc072640 msgr2=0x7f9efc073d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:17.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 shutdown_connections 2024-04-16T07:18:17.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:17.080+0000 7f9f02200700 1 -- 172.21.15.26:0/528200157 wait complete. 2024-04-16T07:18:17.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:17 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/528200157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:17.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:17 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/528200157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T07:18:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:17 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/528200157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:17 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/528200157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T07:18:17.881 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:18:17.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-16T07:18:17.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:18:18.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T07:18:18.526 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi104/config 2024-04-16T07:18:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:18 smithi026 ceph-mon[28805]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:18 smithi104 ceph-mon[36163]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:20 smithi104 ceph-mon[36163]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 -- 172.21.15.104:0/1806290029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84103290 msgr2=0x7fde841036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 --2- 172.21.15.104:0/1806290029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84103290 0x7fde841036a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fde78004940 tx=0x7fde78009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 -- 172.21.15.104:0/1806290029 shutdown_connections 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 --2- 172.21.15.104:0/1806290029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84104540 0x7fde841049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 --2- 172.21.15.104:0/1806290029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84103290 0x7fde841036a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:20.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 -- 172.21.15.104:0/1806290029 >> 172.21.15.104:0/1806290029 conn(0x7fde84072490 msgr2=0x7fde840748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:20.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 -- 172.21.15.104:0/1806290029 shutdown_connections 2024-04-16T07:18:20.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.760+0000 7fde8a200700 1 -- 172.21.15.104:0/1806290029 wait complete. 2024-04-16T07:18:20.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.761+0000 7fde8a200700 1 Processor -- start 2024-04-16T07:18:20.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde8a200700 1 -- start start 2024-04-16T07:18:20.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:20 smithi026 ceph-mon[28805]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde8a200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84103290 0x7fde84113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde8a200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde8a200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde84114a60 con 0x7fde84103290 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde8a200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde841151a0 con 0x7fde84104540 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde83e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde83e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51876/0 (socket says 172.21.15.104:51876) 2024-04-16T07:18:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde83e00700 1 -- 172.21.15.104:0/2305119573 learned_addr learned my addr 172.21.15.104:0/2305119573 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.762+0000 7fde88e00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84103290 0x7fde84113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde83e00700 1 -- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84103290 msgr2=0x7fde84113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde83e00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84103290 0x7fde84113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde83e00700 1 -- 172.21.15.104:0/2305119573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78005040 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde83e00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c8c0 tx=0x7fde700044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde7000a980 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde7000ce60 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde700115e0 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde84115480 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.763+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde84115a40 con 0x7fde84104540 2024-04-16T07:18:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.765+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde840760b0 con 0x7fde84104540 2024-04-16T07:18:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.765+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fde70012400 con 0x7fde84104540 2024-04-16T07:18:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.767+0000 7fde80c00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 0x7fde6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.767+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fde70086fe0 con 0x7fde84104540 2024-04-16T07:18:20.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.767+0000 7fde88e00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 0x7fde6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:20.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.768+0000 7fde88e00700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 0x7fde6c06b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b3b0 tx=0x7fde78009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:20.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:20.771+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fde70057060 con 0x7fde84104540 2024-04-16T07:18:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.064+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fde8404bed0 con 0x7fde84104540 2024-04-16T07:18:21.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.076+0000 7fde80c00700 1 -- 172.21.15.104:0/2305119573 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fde700126b0 con 0x7fde84104540 2024-04-16T07:18:21.075 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-04-16T07:18:21.075 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA9Jh5muZUMBBAA6NKGCOab7u6uN6964rVFHw== 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 msgr2=0x7fde6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 0x7fde6c06b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b3b0 tx=0x7fde78009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 msgr2=0x7fde84114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c8c0 tx=0x7fde700044c0 comp rx=0 tx=0).stop 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 shutdown_connections 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde6c0692a0 0x7fde6c06b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde84103290 0x7fde84113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 --2- 172.21.15.104:0/2305119573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde84104540 0x7fde84114460 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:21.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 >> 172.21.15.104:0/2305119573 conn(0x7fde84072490 msgr2=0x7fde84073b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:21.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 shutdown_connections 2024-04-16T07:18:21.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:18:21.080+0000 7fde8a200700 1 -- 172.21.15.104:0/2305119573 wait complete. 2024-04-16T07:18:21.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:21 smithi026 ceph-mon[28805]: from='client.? 172.21.15.104:0/2305119573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:21.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:21 smithi026 ceph-mon[28805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:21.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:21 smithi026 ceph-mon[28805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T07:18:21.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:21 smithi104 ceph-mon[36163]: from='client.? 172.21.15.104:0/2305119573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:21.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:21 smithi104 ceph-mon[36163]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T07:18:21.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:21 smithi104 ceph-mon[36163]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T07:18:21.929 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-16T07:18:21.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-16T07:18:21.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T07:18:22.082 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:18:22.082 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:18:22.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-16T07:18:22.581 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:22 smithi026 ceph-mon[28805]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:22 smithi104 ceph-mon[36163]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:24.600 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:24 smithi026 ceph-mon[28805]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.662+0000 7f2ee8a00700 1 -- 172.21.15.26:0/1785935960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 msgr2=0x7f2ee41038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.662+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/1785935960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee41038d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0004940 tx=0x7f2ed0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 -- 172.21.15.26:0/1785935960 shutdown_connections 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/1785935960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee41038d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/1785935960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee4109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 -- 172.21.15.26:0/1785935960 >> 172.21.15.26:0/1785935960 conn(0x7f2ee4072c30 msgr2=0x7f2ee4075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 -- 172.21.15.26:0/1785935960 shutdown_connections 2024-04-16T07:18:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.663+0000 7f2ee8a00700 1 -- 172.21.15.26:0/1785935960 wait complete. 2024-04-16T07:18:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 Processor -- start 2024-04-16T07:18:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 -- start start 2024-04-16T07:18:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee411a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee411aaf0 con 0x7f2ee4103460 2024-04-16T07:18:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.664+0000 7f2ee8a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee4114110 con 0x7f2ee41094c0 2024-04-16T07:18:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60624/0 (socket says 172.21.15.26:60624) 2024-04-16T07:18:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 -- 172.21.15.26:0/2022637635 learned_addr learned my addr 172.21.15.26:0/2022637635 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1400700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee411a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 -- 172.21.15.26:0/2022637635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 msgr2=0x7f2ee411a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee411a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1e00700 1 -- 172.21.15.26:0/2022637635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0005040 con 0x7f2ee4103460 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.665+0000 7f2ee1400700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee411a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2ee1e00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c830 tx=0x7f2ed4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed400a760 con 0x7f2ee4103460 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ed400ad90 con 0x7f2ee4103460 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed40113a0 con 0x7f2ee4103460 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee41143f0 con 0x7f2ee4103460 2024-04-16T07:18:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.666+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee41149b0 con 0x7f2ee4103460 2024-04-16T07:18:24.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.667+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee4104ba0 con 0x7f2ee4103460 2024-04-16T07:18:24.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.668+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2ed4012400 con 0x7f2ee4103460 2024-04-16T07:18:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.669+0000 7f2eda000700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 0x7f2ec806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.669+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2ed4086ed0 con 0x7f2ee4103460 2024-04-16T07:18:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.669+0000 7f2ee1400700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 0x7f2ec806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.670+0000 7f2ee1400700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 0x7f2ec806b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4115840 tx=0x7f2ed0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.674+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2ed4053520 con 0x7f2ee4103460 2024-04-16T07:18:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:24 smithi104 ceph-mon[36163]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:24.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.953+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2ee404c710 con 0x7f2ee4103460 2024-04-16T07:18:24.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.964+0000 7f2eda000700 1 -- 172.21.15.26:0/2022637635 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167178 (secure 0 0 0) 0x7f2ed405c020 con 0x7f2ee4103460 2024-04-16T07:18:24.965 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:18:24.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 msgr2=0x7f2ec806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:24.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 0x7f2ec806b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4115840 tx=0x7f2ed0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:24.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 msgr2=0x7f2ee4119f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c830 tx=0x7f2ed4004470 comp rx=0 tx=0).stop 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 shutdown_connections 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2ec80692a0 0x7f2ec806b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ee4103460 0x7f2ee4119f60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 --2- 172.21.15.26:0/2022637635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ee41094c0 0x7f2ee411a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.968+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 >> 172.21.15.26:0/2022637635 conn(0x7f2ee4072c30 msgr2=0x7f2ee4105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.969+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 shutdown_connections 2024-04-16T07:18:24.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:24.969+0000 7f2ee8a00700 1 -- 172.21.15.26:0/2022637635 wait complete. 2024-04-16T07:18:25.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:25 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2022637635' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T07:18:25.728 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"flags":0,"active_gid":14208,"active_name":"smithi026.tawexg","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":3943853548},{"type":"v1","addr":"172.21.15.26:6801","nonce":3943853548}]},"active_addr":"172.21.15.26:6801/3943853548","active_change":"2024-04-16T07:13:31.214991+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi104.uqfljg","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.26:8443/","prometheus":"http://172.21.15.26:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3830655902}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":451733185}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3568818565}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2992109432}]}]} 2024-04-16T07:18:25.732 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-16T07:18:25.732 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-16T07:18:25.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T07:18:25.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:25 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2022637635' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T07:18:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:26.710 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:26 smithi026 ceph-mon[28805]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:26 smithi104 ceph-mon[36163]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:28.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 -- 172.21.15.26:0/1902739441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6901035e0 msgr2=0x7ff6900fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:28.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 --2- 172.21.15.26:0/1902739441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6901035e0 0x7ff6900fcc50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff67c004940 tx=0x7ff67c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:28.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 -- 172.21.15.26:0/1902739441 shutdown_connections 2024-04-16T07:18:28.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 --2- 172.21.15.26:0/1902739441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6901035e0 0x7ff6900fcc50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 --2- 172.21.15.26:0/1902739441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff690102c40 0x7ff690103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 -- 172.21.15.26:0/1902739441 >> 172.21.15.26:0/1902739441 conn(0x7ff6900f8790 msgr2=0x7ff6900faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:28.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 -- 172.21.15.26:0/1902739441 shutdown_connections 2024-04-16T07:18:28.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.421+0000 7ff694e00700 1 -- 172.21.15.26:0/1902739441 wait complete. 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.422+0000 7ff694e00700 1 Processor -- start 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.422+0000 7ff694e00700 1 -- start start 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff694e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff694e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6901035e0 0x7ff690116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff694e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff690116710 con 0x7ff690102c40 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff694e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69010fb50 con 0x7ff6901035e0 2024-04-16T07:18:28.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37708/0 (socket says 172.21.15.26:37708) 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 -- 172.21.15.26:0/3573244507 learned_addr learned my addr 172.21.15.26:0/3573244507 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68ea00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6901035e0 0x7ff690116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 -- 172.21.15.26:0/3573244507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6901035e0 msgr2=0x7ff690116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6901035e0 0x7ff690116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.423+0000 7ff68f400700 1 -- 172.21.15.26:0/3573244507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c005040 con 0x7ff690102c40 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff68f400700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff68000eb40 tx=0x7ff68000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff680007cc0 con 0x7ff690102c40 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff68000f040 con 0x7ff690102c40 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69010fdd0 con 0x7ff690102c40 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff680009620 con 0x7ff690102c40 2024-04-16T07:18:28.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.424+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff690110310 con 0x7ff690102c40 2024-04-16T07:18:28.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.426+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6900fe390 con 0x7ff690102c40 2024-04-16T07:18:28.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.427+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff680009780 con 0x7ff690102c40 2024-04-16T07:18:28.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.428+0000 7ff687400700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 0x7ff67406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:28.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.428+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff680086a70 con 0x7ff690102c40 2024-04-16T07:18:28.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.428+0000 7ff68ea00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 0x7ff67406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:28.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.429+0000 7ff68ea00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 0x7ff67406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00b0d0 tx=0x7ff67c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:28.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.433+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff68004e250 con 0x7ff690102c40 2024-04-16T07:18:28.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.651+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff69004bed0 con 0x7ff690102c40 2024-04-16T07:18:28.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.652+0000 7ff687400700 1 -- 172.21.15.26:0/3573244507 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11085 (secure 0 0 0) 0x7ff6800530f0 con 0x7ff690102c40 2024-04-16T07:18:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:18:28.651 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:18:04.421019+0000","last_up_change":"2024-04-16T07:18:03.334452+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6803","nonce":3225147345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6805","nonce":3225147345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6809","nonce":3225147345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6807","nonce":3225147345}]},"public_addr":"172.21.15.26:6803/3225147345","cluster_addr":"172.21.15.26:6805/3225147345","heartbeat_back_addr":"172.21.15.26:6809/3225147345","heartbeat_front_addr":"172.21.15.26:6807/3225147345","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6811","nonce":1194632492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6813","nonce":1194632492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6817","nonce":1194632492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6815","nonce":1194632492}]},"public_addr":"172.21.15.26:6811/1194632492","cluster_addr":"172.21.15.26:6813/1194632492","heartbeat_back_addr":"172.21.15.26:6817/1194632492","heartbeat_front_addr":"172.21.15.26:6815/1194632492","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6819","nonce":828513924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6821","nonce":828513924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6825","nonce":828513924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6823","nonce":828513924}]},"public_addr":"172.21.15.26:6819/828513924","cluster_addr":"172.21.15.26:6821/828513924","heartbeat_back_addr":"172.21.15.26:6825/828513924","heartbeat_front_addr":"172.21.15.26:6823/828513924","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6801","nonce":2432746791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6803","nonce":2432746791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6807","nonce":2432746791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6805","nonce":2432746791}]},"public_addr":"172.21.15.104:6801/2432746791","cluster_addr":"172.21.15.104:6803/2432746791","heartbeat_back_addr":"172.21.15.104:6807/2432746791","heartbeat_front_addr":"172.21.15.104:6805/2432746791","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6809","nonce":3400087284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6811","nonce":3400087284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6815","nonce":3400087284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6813","nonce":3400087284}]},"public_addr":"172.21.15.104:6809/3400087284","cluster_addr":"172.21.15.104:6811/3400087284","heartbeat_back_addr":"172.21.15.104:6815/3400087284","heartbeat_front_addr":"172.21.15.104:6813/3400087284","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6817","nonce":4159292069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6819","nonce":4159292069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6823","nonce":4159292069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6821","nonce":4159292069}]},"public_addr":"172.21.15.104:6817/4159292069","cluster_addr":"172.21.15.104:6819/4159292069","heartbeat_back_addr":"172.21.15.104:6823/4159292069","heartbeat_front_addr":"172.21.15.104:6821/4159292069","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:18:28.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 msgr2=0x7ff67406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 0x7ff67406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00b0d0 tx=0x7ff67c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 msgr2=0x7ff690115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff68000eb40 tx=0x7ff68000b040 comp rx=0 tx=0).stop 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 shutdown_connections 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff6740692a0 0x7ff67406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff690102c40 0x7ff690115c20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 --2- 172.21.15.26:0/3573244507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6901035e0 0x7ff690116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:28.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 >> 172.21.15.26:0/3573244507 conn(0x7ff6900f8790 msgr2=0x7ff6900703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:28.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 shutdown_connections 2024-04-16T07:18:28.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:28.655+0000 7ff694e00700 1 -- 172.21.15.26:0/3573244507 wait complete. 2024-04-16T07:18:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:28 smithi104 ceph-mon[36163]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:28 smithi026 ceph-mon[28805]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:29.462 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-16T07:18:29.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T07:18:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:29 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3573244507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:29.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:29 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3573244507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:29.996 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:30 smithi026 ceph-mon[28805]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:30 smithi104 ceph-mon[36163]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:31.923 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 -- 172.21.15.26:0/310298715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f00656b0 msgr2=0x7fb4f0065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/310298715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f00656b0 0x7fb4f0065b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc004940 tx=0x7fb4dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 -- 172.21.15.26:0/310298715 shutdown_connections 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/310298715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f00656b0 0x7fb4f0065b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/310298715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f0102e60 0x7fb4f0103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 -- 172.21.15.26:0/310298715 >> 172.21.15.26:0/310298715 conn(0x7fb4f0072bc0 msgr2=0x7fb4f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 -- 172.21.15.26:0/310298715 shutdown_connections 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.232+0000 7fb4f7e00700 1 -- 172.21.15.26:0/310298715 wait complete. 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 Processor -- start 2024-04-16T07:18:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 -- start start 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 0x7fb4f0115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f01168f0 con 0x7fb4f0102e60 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.233+0000 7fb4f7e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f010fcc0 con 0x7fb4f00656b0 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37718/0 (socket says 172.21.15.26:37718) 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 -- 172.21.15.26:0/1602566507 learned_addr learned my addr 172.21.15.26:0/1602566507 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f5400700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 0x7fb4f0115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 -- 172.21.15.26:0/1602566507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 msgr2=0x7fb4f0115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 0x7fb4f0115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f4a00700 1 -- 172.21.15.26:0/1602566507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc005040 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.234+0000 7fb4f5400700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 0x7fb4f0115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4f4a00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009180 tx=0x7fb4dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4dc00e040 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4dc00bbc0 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f010ff40 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4dc01c630 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.235+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f0110500 con 0x7fb4f0102e60 2024-04-16T07:18:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.236+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d00052b0 con 0x7fb4f0102e60 2024-04-16T07:18:32.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.237+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb4dc01c860 con 0x7fb4f0102e60 2024-04-16T07:18:32.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.238+0000 7fb4ed600700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 0x7fb4d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:32.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.238+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4dc088200 con 0x7fb4f0102e60 2024-04-16T07:18:32.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.239+0000 7fb4f5400700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 0x7fb4d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:32.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.239+0000 7fb4f5400700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 0x7fb4d806b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4009fd0 tx=0x7fb4e4009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:32.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.243+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb4dc0582d0 con 0x7fb4f0102e60 2024-04-16T07:18:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.457+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4d0005560 con 0x7fb4f0102e60 2024-04-16T07:18:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.459+0000 7fb4ed600700 1 -- 172.21.15.26:0/1602566507 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11085 (secure 0 0 0) 0x7fb4dc057e60 con 0x7fb4f0102e60 2024-04-16T07:18:32.458 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:18:32.458 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:18:04.421019+0000","last_up_change":"2024-04-16T07:18:03.334452+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6803","nonce":3225147345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6805","nonce":3225147345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6809","nonce":3225147345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6807","nonce":3225147345}]},"public_addr":"172.21.15.26:6803/3225147345","cluster_addr":"172.21.15.26:6805/3225147345","heartbeat_back_addr":"172.21.15.26:6809/3225147345","heartbeat_front_addr":"172.21.15.26:6807/3225147345","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6811","nonce":1194632492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6813","nonce":1194632492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6817","nonce":1194632492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6815","nonce":1194632492}]},"public_addr":"172.21.15.26:6811/1194632492","cluster_addr":"172.21.15.26:6813/1194632492","heartbeat_back_addr":"172.21.15.26:6817/1194632492","heartbeat_front_addr":"172.21.15.26:6815/1194632492","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6819","nonce":828513924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6821","nonce":828513924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6825","nonce":828513924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6823","nonce":828513924}]},"public_addr":"172.21.15.26:6819/828513924","cluster_addr":"172.21.15.26:6821/828513924","heartbeat_back_addr":"172.21.15.26:6825/828513924","heartbeat_front_addr":"172.21.15.26:6823/828513924","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6801","nonce":2432746791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6803","nonce":2432746791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6807","nonce":2432746791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6805","nonce":2432746791}]},"public_addr":"172.21.15.104:6801/2432746791","cluster_addr":"172.21.15.104:6803/2432746791","heartbeat_back_addr":"172.21.15.104:6807/2432746791","heartbeat_front_addr":"172.21.15.104:6805/2432746791","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6809","nonce":3400087284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6811","nonce":3400087284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6815","nonce":3400087284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6813","nonce":3400087284}]},"public_addr":"172.21.15.104:6809/3400087284","cluster_addr":"172.21.15.104:6811/3400087284","heartbeat_back_addr":"172.21.15.104:6815/3400087284","heartbeat_front_addr":"172.21.15.104:6813/3400087284","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6817","nonce":4159292069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6819","nonce":4159292069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6823","nonce":4159292069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6821","nonce":4159292069}]},"public_addr":"172.21.15.104:6817/4159292069","cluster_addr":"172.21.15.104:6819/4159292069","heartbeat_back_addr":"172.21.15.104:6823/4159292069","heartbeat_front_addr":"172.21.15.104:6821/4159292069","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:18:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 msgr2=0x7fb4d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 0x7fb4d806b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4009fd0 tx=0x7fb4e4009ee0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 msgr2=0x7fb4f01162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009180 tx=0x7fb4dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 shutdown_connections 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb4d80692b0 0x7fb4d806b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f00656b0 0x7fb4f0115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 --2- 172.21.15.26:0/1602566507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f0102e60 0x7fb4f01162a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 >> 172.21.15.26:0/1602566507 conn(0x7fb4f0072bc0 msgr2=0x7fb4f00fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.462+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 shutdown_connections 2024-04-16T07:18:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:32.463+0000 7fb4f7e00700 1 -- 172.21.15.26:0/1602566507 wait complete. 2024-04-16T07:18:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:32 smithi104 ceph-mon[36163]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:32 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1602566507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:32.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:32 smithi026 ceph-mon[28805]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:32.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:32 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1602566507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:18:33.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-16T07:18:33.211 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-16T07:18:33.211 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-16T07:18:33.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-16T07:18:33.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-16T07:18:33.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-16T07:18:33.840 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:33.844 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:34 smithi104 ceph-mon[36163]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:34.853 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:34 smithi026 ceph-mon[28805]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:36.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:36 smithi026 ceph-mon[28805]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:36 smithi104 ceph-mon[36163]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:38.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.774+0000 7fed35c00700 1 -- 172.21.15.26:0/3388149506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 msgr2=0x7fed30106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:38.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.774+0000 7fed35c00700 1 --2- 172.21.15.26:0/3388149506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed30106e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fed1c004940 tx=0x7fed1c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:38.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 -- 172.21.15.26:0/3388149506 shutdown_connections 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 --2- 172.21.15.26:0/3388149506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed30106e70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 --2- 172.21.15.26:0/3388149506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 0x7fed300fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 -- 172.21.15.26:0/3388149506 >> 172.21.15.26:0/3388149506 conn(0x7fed300f7900 msgr2=0x7fed300f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 -- 172.21.15.26:0/3388149506 shutdown_connections 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.775+0000 7fed35c00700 1 -- 172.21.15.26:0/3388149506 wait complete. 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 Processor -- start 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 -- start start 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 0x7fed30115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30116710 con 0x7fed300fe830 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.776+0000 7fed35c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3010fd40 con 0x7fed300fde90 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2ea00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 0x7fed30115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37728/0 (socket says 172.21.15.26:37728) 2024-04-16T07:18:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 -- 172.21.15.26:0/3989635963 learned_addr learned my addr 172.21.15.26:0/3989635963 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 -- 172.21.15.26:0/3989635963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 msgr2=0x7fed30115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 0x7fed30115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c005040 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.777+0000 7fed2e000700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009180 tx=0x7fed1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.778+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c00e040 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.778+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed1c013420 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.778+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c01c630 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.778+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3010ffc0 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.778+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30110500 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.779+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed301109e0 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.780+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fed1c013590 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.781+0000 7fed26a00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 0x7fed1406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.781+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fed1c088060 con 0x7fed300fe830 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.781+0000 7fed26a00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 0x7fed14071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.782+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed14071a30 con 0x7fed1406ef70 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.782+0000 7fed2ea00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 0x7fed1406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.782+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed1c08a6b0 con 0x7fed300fe830 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.782+0000 7fed2f400700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 0x7fed14071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.782+0000 7fed2ea00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 0x7fed1406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed20001450 tx=0x7fed2000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.783+0000 7fed2f400700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 0x7fed14071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.784+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== osd.2 v2:172.21.15.26:6818/828513924 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fed14071a30 con 0x7fed1406ef70 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.801+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 --> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fed3005e6c0 con 0x7fed1406ef70 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.801+0000 7fed26a00700 1 -- 172.21.15.26:0/3989635963 <== osd.2 v2:172.21.15.26:6818/828513924 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fed3005e6c0 con 0x7fed1406ef70 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 msgr2=0x7fed14071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 0x7fed14071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 msgr2=0x7fed1406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 0x7fed1406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed20001450 tx=0x7fed2000b040 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 msgr2=0x7fed301160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009180 tx=0x7fed1c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 shutdown_connections 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6818/828513924,v1:172.21.15.26:6819/828513924] conn(0x7fed1406ef70 0x7fed14071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fed140692a0 0x7fed1406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed300fde90 0x7fed30115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 --2- 172.21.15.26:0/3989635963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed300fe830 0x7fed301160c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 >> 172.21.15.26:0/3989635963 conn(0x7fed300f7900 msgr2=0x7fed300fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 shutdown_connections 2024-04-16T07:18:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:37.802+0000 7fed35c00700 1 -- 172.21.15.26:0/3989635963 wait complete. 2024-04-16T07:18:38.706 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:38 smithi026 ceph-mon[28805]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:38 smithi104 ceph-mon[36163]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:40.233 INFO:teuthology.orchestra.run.smithi026.stdout:81604378654 2024-04-16T07:18:40.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-16T07:18:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:40 smithi026 ceph-mon[28805]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:40 smithi104 ceph-mon[36163]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:42.652 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:42 smithi026 ceph-mon[28805]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:42 smithi104 ceph-mon[36163]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 -- 172.21.15.26:0/2484777281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f38102e20 msgr2=0x7f8f381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 --2- 172.21.15.26:0/2484777281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f38102e20 0x7f8f381031f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c004940 tx=0x7f8f2c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 -- 172.21.15.26:0/2484777281 shutdown_connections 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 --2- 172.21.15.26:0/2484777281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f381037c0 0x7f8f380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 --2- 172.21.15.26:0/2484777281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f38102e20 0x7f8f381031f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 -- 172.21.15.26:0/2484777281 >> 172.21.15.26:0/2484777281 conn(0x7f8f380f89d0 msgr2=0x7f8f380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 -- 172.21.15.26:0/2484777281 shutdown_connections 2024-04-16T07:18:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.156+0000 7f8f40400700 1 -- 172.21.15.26:0/2484777281 wait complete. 2024-04-16T07:18:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.157+0000 7f8f40400700 1 Processor -- start 2024-04-16T07:18:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.157+0000 7f8f40400700 1 -- start start 2024-04-16T07:18:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f40400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f38102e20 0x7f8f38115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f40400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f40400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38116910 con 0x7f8f381037c0 2024-04-16T07:18:43.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f40400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3810fd20 con 0x7f8f38102e20 2024-04-16T07:18:43.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f3d000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f3d000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60766/0 (socket says 172.21.15.26:60766) 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f3d000700 1 -- 172.21.15.26:0/1511111999 learned_addr learned my addr 172.21.15.26:0/1511111999 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.158+0000 7f8f3da00700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f38102e20 0x7f8f38115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f3d000700 1 -- 172.21.15.26:0/1511111999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f38102e20 msgr2=0x7f8f38115d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f3d000700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f38102e20 0x7f8f38115d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f3d000700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c005040 con 0x7f8f381037c0 2024-04-16T07:18:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f3d000700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400c800 tx=0x7f8f24004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f2400a760 con 0x7f8f381037c0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f2400ad90 con 0x7f8f381037c0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f240113a0 con 0x7f8f381037c0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.159+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f38110000 con 0x7f8f381037c0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.160+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f381105c0 con 0x7f8f381037c0 2024-04-16T07:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.161+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f38110b00 con 0x7f8f381037c0 2024-04-16T07:18:43.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.162+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8f24012400 con 0x7f8f381037c0 2024-04-16T07:18:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.163+0000 7f8f35600700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 0x7f8f2006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.163+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8f24086f80 con 0x7f8f381037c0 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.163+0000 7f8f3da00700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 0x7f8f2006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.164+0000 7f8f35600700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 0x7f8f20071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.164+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f20071a40 con 0x7f8f2006ef80 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.164+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f240894d0 con 0x7f8f381037c0 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.164+0000 7f8f3e400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 0x7f8f20071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.164+0000 7f8f3da00700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 0x7f8f2006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b020 tx=0x7f8f2c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.165+0000 7f8f3e400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 0x7f8f20071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.166+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== osd.1 v2:172.21.15.26:6810/1194632492 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8f20071a40 con 0x7f8f2006ef80 2024-04-16T07:18:43.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.183+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 --> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8f3805e6c0 con 0x7f8f2006ef80 2024-04-16T07:18:43.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.184+0000 7f8f35600700 1 -- 172.21.15.26:0/1511111999 <== osd.1 v2:172.21.15.26:6810/1194632492 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8f3805e6c0 con 0x7f8f2006ef80 2024-04-16T07:18:43.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.184+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 msgr2=0x7f8f20071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.184+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 0x7f8f20071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 msgr2=0x7f8f2006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 0x7f8f2006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b020 tx=0x7f8f2c009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 msgr2=0x7f8f381162c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400c800 tx=0x7f8f24004470 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 shutdown_connections 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6810/1194632492,v1:172.21.15.26:6811/1194632492] conn(0x7f8f2006ef80 0x7f8f20071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f8f200692b0 0x7f8f2006b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f38102e20 0x7f8f38115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 --2- 172.21.15.26:0/1511111999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f381037c0 0x7f8f381162c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 >> 172.21.15.26:0/1511111999 conn(0x7f8f380f89d0 msgr2=0x7f8f38070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 shutdown_connections 2024-04-16T07:18:43.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.185+0000 7f8f40400700 1 -- 172.21.15.26:0/1511111999 wait complete. 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 -- 172.21.15.26:0/2258587797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fe7b0 msgr2=0x7f7838106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 --2- 172.21.15.26:0/2258587797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fe7b0 0x7f7838106df0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7824004940 tx=0x7f7824009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 -- 172.21.15.26:0/2258587797 shutdown_connections 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 --2- 172.21.15.26:0/2258587797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fe7b0 0x7f7838106df0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 --2- 172.21.15.26:0/2258587797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fde10 0x7f78380fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 -- 172.21.15.26:0/2258587797 >> 172.21.15.26:0/2258587797 conn(0x7f78380f7880 msgr2=0x7f78380f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 -- 172.21.15.26:0/2258587797 shutdown_connections 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.086+0000 7f783de00700 1 -- 172.21.15.26:0/2258587797 wait complete. 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.087+0000 7f783de00700 1 Processor -- start 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.087+0000 7f783de00700 1 -- start start 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f783de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f783de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fe7b0 0x7f7838116330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f783de00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838116980 con 0x7f78380fde10 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f783de00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783810fd50 con 0x7f78380fe7b0 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f7836a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f7836a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60762/0 (socket says 172.21.15.26:60762) 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f7836a00700 1 -- 172.21.15.26:0/3177160811 learned_addr learned my addr 172.21.15.26:0/3177160811 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.088+0000 7f7836000700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fe7b0 0x7f7838116330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f7836a00700 1 -- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fe7b0 msgr2=0x7f7838116330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f7836a00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fe7b0 0x7f7838116330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f7836a00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824005040 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f7836a00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7828002a50 tx=0x7f78280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7828007cc0 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f782800f040 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7828009640 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7838110030 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.089+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7838110570 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.090+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7838110a20 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.092+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f78280097a0 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.092+0000 7f782ea00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 0x7f781c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.093+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7828087690 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.093+0000 7f7836000700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 0x7f781c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.093+0000 7f782ea00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 0x7f781c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.093+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f781c071a40 con 0x7f781c06ef80 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.093+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78280879d0 con 0x7f78380fde10 2024-04-16T07:18:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.094+0000 7f7837400700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 0x7f781c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.094+0000 7f7836000700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 0x7f781c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7838111690 tx=0x7f7824009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.095+0000 7f7837400700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 0x7f781c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.097+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== osd.5 v2:172.21.15.104:6816/4159292069 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f781c071a40 con 0x7f781c06ef80 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.114+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 --> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7838110be0 con 0x7f781c06ef80 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.115+0000 7f782ea00700 1 -- 172.21.15.26:0/3177160811 <== osd.5 v2:172.21.15.104:6816/4159292069 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f783000a8d0 con 0x7f781c06ef80 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.115+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 msgr2=0x7f781c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.115+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 0x7f781c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 msgr2=0x7f781c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 0x7f781c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7838111690 tx=0x7f7824009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 msgr2=0x7f7838115df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7828002a50 tx=0x7f78280046b0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 shutdown_connections 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:6816/4159292069,v1:172.21.15.104:6817/4159292069] conn(0x7f781c06ef80 0x7f781c071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f781c0692b0 0x7f781c06b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78380fde10 0x7f7838115df0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 --2- 172.21.15.26:0/3177160811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78380fe7b0 0x7f7838116330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 >> 172.21.15.26:0/3177160811 conn(0x7f78380f7880 msgr2=0x7f78380f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 shutdown_connections 2024-04-16T07:18:43.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.116+0000 7f783de00700 1 -- 172.21.15.26:0/3177160811 wait complete. 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 -- 172.21.15.26:0/3407929827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 msgr2=0x7f01280fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 --2- 172.21.15.26:0/3407929827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f01280fe040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0118004940 tx=0x7f0118009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 -- 172.21.15.26:0/3407929827 shutdown_connections 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 --2- 172.21.15.26:0/3407929827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f0128106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 --2- 172.21.15.26:0/3407929827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f01280fe040 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 -- 172.21.15.26:0/3407929827 >> 172.21.15.26:0/3407929827 conn(0x7f01280f7700 msgr2=0x7f01280f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.984+0000 7f012d800700 1 -- 172.21.15.26:0/3407929827 shutdown_connections 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.985+0000 7f012d800700 1 -- 172.21.15.26:0/3407929827 wait complete. 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 Processor -- start 2024-04-16T07:18:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 -- start start 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f01281160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01281167d0 con 0x7f01280fdc70 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.986+0000 7f012d800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012810fb50 con 0x7f01280fe610 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60736/0 (socket says 172.21.15.26:60736) 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 -- 172.21.15.26:0/4087855974 learned_addr learned my addr 172.21.15.26:0/4087855974 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127400700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f01281160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 -- 172.21.15.26:0/4087855974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 msgr2=0x7f01281160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f01281160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118005040 con 0x7f01280fdc70 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127400700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f01281160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.987+0000 7f0127e00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0118009180 tx=0x7f0118009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.988+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f011800e040 con 0x7f01280fdc70 2024-04-16T07:18:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.988+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0118013420 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.988+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f011801c630 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.988+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012810fdd0 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.988+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0128110310 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.989+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0128110940 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.990+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0118013590 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f011fe00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 0x7f010c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f01180881e0 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f0127400700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 0x7f010c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f011fe00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 0x7f010c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f010c071a40 con 0x7f010c06ef80 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.991+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f011808a730 con 0x7f01280fdc70 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.992+0000 7f012ce00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 0x7f010c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.992+0000 7f0127400700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 0x7f010c06b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0114006fd0 tx=0x7f0114008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.993+0000 7f012ce00700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 0x7f010c071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:42.994+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== osd.0 v2:172.21.15.26:6802/3225147345 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f010c071a40 con 0x7f010c06ef80 2024-04-16T07:18:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.011+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 --> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f012810ff60 con 0x7f010c06ef80 2024-04-16T07:18:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f011fe00700 1 -- 172.21.15.26:0/4087855974 <== osd.0 v2:172.21.15.26:6802/3225147345 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f012810ff60 con 0x7f010c06ef80 2024-04-16T07:18:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 msgr2=0x7f010c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 0x7f010c071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 msgr2=0x7f010c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 0x7f010c06b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0114006fd0 tx=0x7f0114008040 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.012+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 msgr2=0x7f0128115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0118009180 tx=0x7f0118009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 shutdown_connections 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6802/3225147345,v1:172.21.15.26:6803/3225147345] conn(0x7f010c06ef80 0x7f010c071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f010c0692b0 0x7f010c06b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01280fdc70 0x7f0128115bb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 --2- 172.21.15.26:0/4087855974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01280fe610 0x7f01281160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 >> 172.21.15.26:0/4087855974 conn(0x7f01280f7700 msgr2=0x7f01280fbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 shutdown_connections 2024-04-16T07:18:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:43.013+0000 7f012d800700 1 -- 172.21.15.26:0/4087855974 wait complete. 2024-04-16T07:18:43.831 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:43 smithi026 ceph-mon[28805]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:44.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:43 smithi104 ceph-mon[36163]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:44.490 INFO:teuthology.orchestra.run.smithi026.stdout:38654705709 2024-04-16T07:18:44.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-16T07:18:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 -- 172.21.15.26:0/3202774255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f381037c0 msgr2=0x7f9f380fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 --2- 172.21.15.26:0/3202774255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f381037c0 0x7f9f380fce30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9f24004940 tx=0x7f9f24009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 -- 172.21.15.26:0/3202774255 shutdown_connections 2024-04-16T07:18:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 --2- 172.21.15.26:0/3202774255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f381037c0 0x7f9f380fce30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 --2- 172.21.15.26:0/3202774255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f38102e20 0x7f9f381031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.684+0000 7f9f3f600700 1 -- 172.21.15.26:0/3202774255 >> 172.21.15.26:0/3202774255 conn(0x7f9f380f89d0 msgr2=0x7f9f380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.686+0000 7f9f3f600700 1 -- 172.21.15.26:0/3202774255 shutdown_connections 2024-04-16T07:18:44.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.686+0000 7f9f3f600700 1 -- 172.21.15.26:0/3202774255 wait complete. 2024-04-16T07:18:44.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.687+0000 7f9f3f600700 1 Processor -- start 2024-04-16T07:18:44.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.687+0000 7f9f3f600700 1 -- start start 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3f600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3f600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 0x7f9f38116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3f600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38116970 con 0x7f9f38102e20 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3f600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3810fd40 con 0x7f9f381037c0 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3cc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3cc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60778/0 (socket says 172.21.15.26:60778) 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f3cc00700 1 -- 172.21.15.26:0/2947331913 learned_addr learned my addr 172.21.15.26:0/2947331913 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.688+0000 7f9f37e00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 0x7f9f38116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f3cc00700 1 -- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 msgr2=0x7f9f38116320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f3cc00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 0x7f9f38116320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f3cc00700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f24005040 con 0x7f9f38102e20 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f37e00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 0x7f9f38116320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f3cc00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c002a50 tx=0x7f9f2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2c007cc0 con 0x7f9f38102e20 2024-04-16T07:18:44.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f2c00f040 con 0x7f9f38102e20 2024-04-16T07:18:44.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.689+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f38110020 con 0x7f9f38102e20 2024-04-16T07:18:44.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.690+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2c009640 con 0x7f9f38102e20 2024-04-16T07:18:44.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.690+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f381105e0 con 0x7f9f38102e20 2024-04-16T07:18:44.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.691+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f38110ac0 con 0x7f9f38102e20 2024-04-16T07:18:44.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.692+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9f2c0097a0 con 0x7f9f38102e20 2024-04-16T07:18:44.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.693+0000 7f9f34c00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 0x7f9f2006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:44.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.693+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9f2c086c50 con 0x7f9f38102e20 2024-04-16T07:18:44.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f37e00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 0x7f9f2006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:44.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f34c00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 0x7f9f20071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:44.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f20071a40 con 0x7f9f2006ef80 2024-04-16T07:18:44.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f2c08c2b0 con 0x7f9f38102e20 2024-04-16T07:18:44.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f37e00700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 0x7f9f2006b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f38111680 tx=0x7f9f24009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:44.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.694+0000 7f9f3d600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 0x7f9f20071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:44.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.695+0000 7f9f3d600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 0x7f9f20071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:44.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.697+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== osd.4 v2:172.21.15.104:6808/3400087284 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9f20071a40 con 0x7f9f2006ef80 2024-04-16T07:18:44.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.714+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 --> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f380fe570 con 0x7f9f2006ef80 2024-04-16T07:18:44.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.714+0000 7f9f34c00700 1 -- 172.21.15.26:0/2947331913 <== osd.4 v2:172.21.15.104:6808/3400087284 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9f380fe570 con 0x7f9f2006ef80 2024-04-16T07:18:44.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 msgr2=0x7f9f20071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:44.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 0x7f9f20071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 msgr2=0x7f9f2006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 0x7f9f2006b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f38111680 tx=0x7f9f24009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 msgr2=0x7f9f38115de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c002a50 tx=0x7f9f2c0046b0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 shutdown_connections 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:6808/3400087284,v1:172.21.15.104:6809/3400087284] conn(0x7f9f2006ef80 0x7f9f20071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9f200692b0 0x7f9f2006b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.715+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f38102e20 0x7f9f38115de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.716+0000 7f9f3f600700 1 --2- 172.21.15.26:0/2947331913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f381037c0 0x7f9f38116320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.716+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 >> 172.21.15.26:0/2947331913 conn(0x7f9f380f89d0 msgr2=0x7f9f381021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.716+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 shutdown_connections 2024-04-16T07:18:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:44.716+0000 7f9f3f600700 1 -- 172.21.15.26:0/2947331913 wait complete. 2024-04-16T07:18:45.588 INFO:teuthology.orchestra.run.smithi026.stdout:60129542181 2024-04-16T07:18:45.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-16T07:18:46.170 INFO:teuthology.orchestra.run.smithi026.stdout:171798691850 2024-04-16T07:18:46.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-16T07:18:46.529 INFO:teuthology.orchestra.run.smithi026.stdout:150323855378 2024-04-16T07:18:46.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-16T07:18:46.688 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:46 smithi026 ceph-mon[28805]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:46.688 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:46.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 -- 172.21.15.26:0/3201045546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 msgr2=0x7f438c103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 --2- 172.21.15.26:0/3201045546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 0x7f438c103230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4378004880 tx=0x7f4378009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 -- 172.21.15.26:0/3201045546 shutdown_connections 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 --2- 172.21.15.26:0/3201045546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 --2- 172.21.15.26:0/3201045546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 0x7f438c103230 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 -- 172.21.15.26:0/3201045546 >> 172.21.15.26:0/3201045546 conn(0x7f438c072bc0 msgr2=0x7f438c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 -- 172.21.15.26:0/3201045546 shutdown_connections 2024-04-16T07:18:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.748+0000 7f4392200700 1 -- 172.21.15.26:0/3201045546 wait complete. 2024-04-16T07:18:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.749+0000 7f4392200700 1 Processor -- start 2024-04-16T07:18:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.750+0000 7f4392200700 1 -- start start 2024-04-16T07:18:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.750+0000 7f4392200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.750+0000 7f4392200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 0x7f438c10bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.750+0000 7f4392200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c10c380 con 0x7f438c0656b0 2024-04-16T07:18:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.750+0000 7f4392200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c10c4c0 con 0x7f438c102e60 2024-04-16T07:18:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.751+0000 7f438b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.751+0000 7f438b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60800/0 (socket says 172.21.15.26:60800) 2024-04-16T07:18:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.751+0000 7f438b400700 1 -- 172.21.15.26:0/1267558317 learned_addr learned my addr 172.21.15.26:0/1267558317 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.751+0000 7f438b400700 1 -- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 msgr2=0x7f438c10bdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-16T07:18:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.751+0000 7f438b400700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 0x7f438c10bdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.752+0000 7f438b400700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378005040 con 0x7f438c0656b0 2024-04-16T07:18:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.752+0000 7f438b400700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43780134a0 tx=0x7f4378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.752+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437800e030 con 0x7f438c0656b0 2024-04-16T07:18:46.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.753+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c11e250 con 0x7f438c0656b0 2024-04-16T07:18:46.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.753+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c11e6d0 con 0x7f438c0656b0 2024-04-16T07:18:46.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.753+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4378013d60 con 0x7f438c0656b0 2024-04-16T07:18:46.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.753+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437801cd60 con 0x7f438c0656b0 2024-04-16T07:18:46.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.754+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f436c000fd0 con 0x7f438c0656b0 2024-04-16T07:18:46.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.756+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f437802f430 con 0x7f438c0656b0 2024-04-16T07:18:46.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f4383400700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 0x7f437006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:46.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4378089000 con 0x7f438c0656b0 2024-04-16T07:18:46.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f4383400700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 0x7f4370071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:46.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4370071a30 con 0x7f437006ef70 2024-04-16T07:18:46.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f438aa00700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 0x7f437006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:46.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.757+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43780892e0 con 0x7f438c0656b0 2024-04-16T07:18:46.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.758+0000 7f438be00700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 0x7f4370071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:46.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.758+0000 7f438aa00700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 0x7f437006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f438c10d1f0 tx=0x7f437c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:46.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.759+0000 7f438be00700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 0x7f4370071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.761+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== osd.3 v2:172.21.15.104:6800/2432746791 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4370071a30 con 0x7f437006ef70 2024-04-16T07:18:46.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:46 smithi104 ceph-mon[36163]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:46.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:18:46.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.789+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 --> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f436c002d30 con 0x7f437006ef70 2024-04-16T07:18:46.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.790+0000 7f4383400700 1 -- 172.21.15.26:0/1267558317 <== osd.3 v2:172.21.15.104:6800/2432746791 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f436c002d30 con 0x7f437006ef70 2024-04-16T07:18:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 msgr2=0x7f4370071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 0x7f4370071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 msgr2=0x7f437006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 0x7f437006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f438c10d1f0 tx=0x7f437c00b040 comp rx=0 tx=0).stop 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 msgr2=0x7f438c10b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43780134a0 tx=0x7f4378009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 shutdown_connections 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:6800/2432746791,v1:172.21.15.104:6801/2432746791] conn(0x7f437006ef70 0x7f4370071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43700692a0 0x7f437006b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f438c0656b0 0x7f438c10b8b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 --2- 172.21.15.26:0/1267558317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f438c102e60 0x7f438c10bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:46.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 >> 172.21.15.26:0/1267558317 conn(0x7f438c072bc0 msgr2=0x7f438c0fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:46.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 shutdown_connections 2024-04-16T07:18:46.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:46.791+0000 7f4392200700 1 -- 172.21.15.26:0/1267558317 wait complete. 2024-04-16T07:18:47.055 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:47.902 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:48.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:48 smithi026 ceph-mon[28805]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:48.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:48 smithi104 ceph-mon[36163]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:48.967 INFO:teuthology.orchestra.run.smithi026.stdout:115964117017 2024-04-16T07:18:48.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-16T07:18:49.046 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:49.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.732+0000 7fbad7200700 1 -- 172.21.15.26:0/2562567421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad0105030 msgr2=0x7fbad0105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.732+0000 7fbad7200700 1 --2- 172.21.15.26:0/2562567421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad0105030 0x7fbad0105400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbac0004940 tx=0x7fbac0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 -- 172.21.15.26:0/2562567421 shutdown_connections 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 --2- 172.21.15.26:0/2562567421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad01059d0 0x7fbad00ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 --2- 172.21.15.26:0/2562567421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad0105030 0x7fbad0105400 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 -- 172.21.15.26:0/2562567421 >> 172.21.15.26:0/2562567421 conn(0x7fbad00fabd0 msgr2=0x7fbad00fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 -- 172.21.15.26:0/2562567421 shutdown_connections 2024-04-16T07:18:49.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.733+0000 7fbad7200700 1 -- 172.21.15.26:0/2562567421 wait complete. 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 Processor -- start 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 -- start start 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad0105030 0x7fbad0115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0116780 con 0x7fbad01059d0 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.734+0000 7fbad7200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad010fd60 con 0x7fbad0105030 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38994/0 (socket says 172.21.15.26:38994) 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 -- 172.21.15.26:0/376647899 learned_addr learned my addr 172.21.15.26:0/376647899 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacfe00700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad0105030 0x7fbad0115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 -- 172.21.15.26:0/376647899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad0105030 msgr2=0x7fbad0115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:49.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad0105030 0x7fbad0115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 -- 172.21.15.26:0/376647899 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0005040 con 0x7fbad01059d0 2024-04-16T07:18:49.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.735+0000 7fbacf400700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbabc002a80 tx=0x7fbabc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:49.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.736+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc007cc0 con 0x7fbad01059d0 2024-04-16T07:18:49.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.736+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbabc00f040 con 0x7fbad01059d0 2024-04-16T07:18:49.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.736+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad0110040 con 0x7fbad01059d0 2024-04-16T07:18:49.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.736+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc00daf0 con 0x7fbad01059d0 2024-04-16T07:18:49.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.736+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad0110600 con 0x7fbad01059d0 2024-04-16T07:18:49.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.737+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad0100780 con 0x7fbad01059d0 2024-04-16T07:18:49.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.738+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fbabc00dc90 con 0x7fbad01059d0 2024-04-16T07:18:49.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.740+0000 7fbac7e00700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 0x7fbab406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.740+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbabc086560 con 0x7fbad01059d0 2024-04-16T07:18:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.740+0000 7fbacfe00700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 0x7fbab406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.741+0000 7fbacfe00700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 0x7fbab406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbac000b3b0 tx=0x7fbac0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.745+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbabc04d250 con 0x7fbad01059d0 2024-04-16T07:18:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.977+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbad010c150 con 0x7fbad01059d0 2024-04-16T07:18:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.978+0000 7fbac7e00700 1 -- 172.21.15.26:0/376647899 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbabc0565c0 con 0x7fbad01059d0 2024-04-16T07:18:49.977 INFO:teuthology.orchestra.run.smithi026.stdout:81604378656 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 msgr2=0x7fbab406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 0x7fbab406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbac000b3b0 tx=0x7fbac0009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 msgr2=0x7fbad0116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbabc002a80 tx=0x7fbabc004630 comp rx=0 tx=0).stop 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 shutdown_connections 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fbab40692a0 0x7fbab406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbad0105030 0x7fbad0115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 --2- 172.21.15.26:0/376647899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbad01059d0 0x7fbad0116130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 >> 172.21.15.26:0/376647899 conn(0x7fbad00fabd0 msgr2=0x7fbad00fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.981+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 shutdown_connections 2024-04-16T07:18:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:49.982+0000 7fbad7200700 1 -- 172.21.15.26:0/376647899 wait complete. 2024-04-16T07:18:50.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:50 smithi026 ceph-mon[28805]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:50.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:50 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/376647899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T07:18:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:50 smithi104 ceph-mon[36163]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:50 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/376647899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T07:18:52.739 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:52 smithi026 ceph-mon[28805]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:52.770 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:52.779 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:18:52.797 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378654 got 81604378656 for osd.2 2024-04-16T07:18:52.797 DEBUG:teuthology.parallel:result is None 2024-04-16T07:18:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:52 smithi104 ceph-mon[36163]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:53.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 -- 172.21.15.26:0/2959637319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 msgr2=0x7f8a14070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 --2- 172.21.15.26:0/2959637319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a14070730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a08004940 tx=0x7f8a08009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 -- 172.21.15.26:0/2959637319 shutdown_connections 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 --2- 172.21.15.26:0/2959637319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1410d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 --2- 172.21.15.26:0/2959637319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a14070730 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.352+0000 7f8a1c600700 1 -- 172.21.15.26:0/2959637319 >> 172.21.15.26:0/2959637319 conn(0x7f8a140f8980 msgr2=0x7f8a140fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.353+0000 7f8a1c600700 1 -- 172.21.15.26:0/2959637319 shutdown_connections 2024-04-16T07:18:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.353+0000 7f8a1c600700 1 -- 172.21.15.26:0/2959637319 wait complete. 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.353+0000 7f8a1c600700 1 Processor -- start 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a1c600700 1 -- start start 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a1c600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a1c600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1411a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a1c600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1411abe0 con 0x7f8a14070360 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a1c600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14114200 con 0x7f8a14070c70 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a19c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a19c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39008/0 (socket says 172.21.15.26:39008) 2024-04-16T07:18:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.354+0000 7f8a19c00700 1 -- 172.21.15.26:0/1485587944 learned_addr learned my addr 172.21.15.26:0/1485587944 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19200700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1411a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19c00700 1 -- 172.21.15.26:0/1485587944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 msgr2=0x7f8a1411a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19c00700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1411a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19c00700 1 -- 172.21.15.26:0/1485587944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08005040 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19200700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1411a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a19c00700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009180 tx=0x7f8a08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a0800e040 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a08013420 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.355+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a14114480 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.356+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a0801c630 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.356+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a141149c0 con 0x7f8a14070360 2024-04-16T07:18:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.357+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1404c710 con 0x7f8a14070360 2024-04-16T07:18:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.358+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8a08013590 con 0x7f8a14070360 2024-04-16T07:18:53.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.359+0000 7f8a11600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 0x7f89fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.359+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a080882c0 con 0x7f8a14070360 2024-04-16T07:18:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.359+0000 7f8a19200700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 0x7f89fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.360+0000 7f8a19200700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 0x7f89fc06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007fd0 tx=0x7f8a00009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.364+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8a08054890 con 0x7f8a14070360 2024-04-16T07:18:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.588+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8a14071aa0 con 0x7f8a14070360 2024-04-16T07:18:53.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.594+0000 7f8a11600700 1 -- 172.21.15.26:0/1485587944 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a08057eb0 con 0x7f8a14070360 2024-04-16T07:18:53.593 INFO:teuthology.orchestra.run.smithi026.stdout:60129542183 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.597+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 msgr2=0x7f89fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.597+0000 7f8a1c600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 0x7f89fc06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007fd0 tx=0x7f8a00009040 comp rx=0 tx=0).stop 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 msgr2=0x7f8a1411a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009180 tx=0x7f8a08009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 shutdown_connections 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89fc0692a0 0x7f89fc06b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14070360 0x7f8a1411a050 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 --2- 172.21.15.26:0/1485587944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a14070c70 0x7f8a1411a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 >> 172.21.15.26:0/1485587944 conn(0x7f8a140f8980 msgr2=0x7f8a140f9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 shutdown_connections 2024-04-16T07:18:53.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:53.598+0000 7f8a1c600700 1 -- 172.21.15.26:0/1485587944 wait complete. 2024-04-16T07:18:53.614 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:53 smithi026 ceph-mon[28805]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:53 smithi104 ceph-mon[36163]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:54 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1485587944' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T07:18:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:54 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1485587944' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T07:18:55.710 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:55 smithi026 ceph-mon[28805]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:56.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:55 smithi104 ceph-mon[36163]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:56.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 -- 172.21.15.26:0/1660140878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 msgr2=0x7f155c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 --2- 172.21.15.26:0/1660140878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c106df0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1548004940 tx=0x7f1548009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 -- 172.21.15.26:0/1660140878 shutdown_connections 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 --2- 172.21.15.26:0/1660140878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c106df0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 --2- 172.21.15.26:0/1660140878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 0x7f155c0fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 -- 172.21.15.26:0/1660140878 >> 172.21.15.26:0/1660140878 conn(0x7f155c0f7880 msgr2=0x7f155c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 -- 172.21.15.26:0/1660140878 shutdown_connections 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.095+0000 7f1561800700 1 -- 172.21.15.26:0/1660140878 wait complete. 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 Processor -- start 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 -- start start 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 0x7f155c115df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c116980 con 0x7f155c0fe7b0 2024-04-16T07:18:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f1561800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c10fd50 con 0x7f155c0fde10 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f155a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f155a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39034/0 (socket says 172.21.15.26:39034) 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.096+0000 7f155a000700 1 -- 172.21.15.26:0/3744179377 learned_addr learned my addr 172.21.15.26:0/3744179377 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.097+0000 7f155aa00700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 0x7f155c115df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.097+0000 7f155a000700 1 -- 172.21.15.26:0/3744179377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 msgr2=0x7f155c115df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.097+0000 7f155a000700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 0x7f155c115df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.097+0000 7f155a000700 1 -- 172.21.15.26:0/3744179377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548005040 con 0x7f155c0fe7b0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.097+0000 7f155a000700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1548009180 tx=0x7f1548009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.098+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f154800e040 con 0x7f155c0fe7b0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.098+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c10ffd0 con 0x7f155c0fe7b0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.098+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1548013420 con 0x7f155c0fe7b0 2024-04-16T07:18:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.098+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f154801c630 con 0x7f155c0fe7b0 2024-04-16T07:18:56.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.098+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c110510 con 0x7f155c0fe7b0 2024-04-16T07:18:56.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.099+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c066780 con 0x7f155c0fe7b0 2024-04-16T07:18:56.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.100+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f154801c8f0 con 0x7f155c0fe7b0 2024-04-16T07:18:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.101+0000 7f1552a00700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 0x7f154006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.101+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1548088250 con 0x7f155c0fe7b0 2024-04-16T07:18:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.101+0000 7f155aa00700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 0x7f154006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.102+0000 7f155aa00700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 0x7f154006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.106+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1548058350 con 0x7f155c0fe7b0 2024-04-16T07:18:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.335+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f155c05f5f0 con 0x7f155c0fe7b0 2024-04-16T07:18:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.336+0000 7f1552a00700 1 -- 172.21.15.26:0/3744179377 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1548057ee0 con 0x7f155c0fe7b0 2024-04-16T07:18:56.335 INFO:teuthology.orchestra.run.smithi026.stdout:38654705712 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 msgr2=0x7f154006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 0x7f154006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c009040 comp rx=0 tx=0).stop 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 msgr2=0x7f155c116330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1548009180 tx=0x7f1548009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 shutdown_connections 2024-04-16T07:18:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f15400692b0 0x7f154006b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f155c0fde10 0x7f155c115df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 --2- 172.21.15.26:0/3744179377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f155c0fe7b0 0x7f155c116330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 >> 172.21.15.26:0/3744179377 conn(0x7f155c0f7880 msgr2=0x7f155c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.339+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 shutdown_connections 2024-04-16T07:18:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.340+0000 7f1561800700 1 -- 172.21.15.26:0/3744179377 wait complete. 2024-04-16T07:18:56.842 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:56 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3744179377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T07:18:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:56 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3744179377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T07:18:57.697 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:57 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2024509758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T07:18:57.697 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:57 smithi026 ceph-mon[28805]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 -- 172.21.15.26:0/1708481667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 msgr2=0x7efe6c1053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 --2- 172.21.15.26:0/1708481667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c1053d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009ce0 tx=0x7efe5c004d10 comp rx=0 tx=0).stop 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 -- 172.21.15.26:0/1708481667 shutdown_connections 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 --2- 172.21.15.26:0/1708481667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 0x7efe6c0ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 --2- 172.21.15.26:0/1708481667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c1053d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 -- 172.21.15.26:0/1708481667 >> 172.21.15.26:0/1708481667 conn(0x7efe6c0fabd0 msgr2=0x7efe6c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.870+0000 7efe71a00700 1 -- 172.21.15.26:0/1708481667 shutdown_connections 2024-04-16T07:18:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.871+0000 7efe71a00700 1 -- 172.21.15.26:0/1708481667 wait complete. 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.872+0000 7efe71a00700 1 Processor -- start 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.872+0000 7efe71a00700 1 -- start start 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe71a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe71a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 0x7efe6c116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe71a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c116760 con 0x7efe6c105000 2024-04-16T07:18:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe71a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c10fd40 con 0x7efe6c1059a0 2024-04-16T07:18:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe6aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe6aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39072/0 (socket says 172.21.15.26:39072) 2024-04-16T07:18:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe6aa00700 1 -- 172.21.15.26:0/4132210667 learned_addr learned my addr 172.21.15.26:0/4132210667 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.873+0000 7efe6a000700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 0x7efe6c116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.874+0000 7efe6aa00700 1 -- 172.21.15.26:0/4132210667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 msgr2=0x7efe6c116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.874+0000 7efe6aa00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 0x7efe6c116110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.874+0000 7efe6aa00700 1 -- 172.21.15.26:0/4132210667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c009940 con 0x7efe6c105000 2024-04-16T07:18:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.874+0000 7efe6aa00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00ba50 tx=0x7efe5c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.874+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe5c00e040 con 0x7efe6c105000 2024-04-16T07:18:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.875+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c10ffc0 con 0x7efe6c105000 2024-04-16T07:18:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.875+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe5c004610 con 0x7efe6c105000 2024-04-16T07:18:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.875+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c110580 con 0x7efe6c105000 2024-04-16T07:18:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.875+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe5c012a00 con 0x7efe6c105000 2024-04-16T07:18:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.876+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c100750 con 0x7efe6c105000 2024-04-16T07:18:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.877+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7efe5c01c650 con 0x7efe6c105000 2024-04-16T07:18:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.878+0000 7efe62a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 0x7efe5006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.878+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efe5c088f10 con 0x7efe6c105000 2024-04-16T07:18:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.879+0000 7efe6a000700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 0x7efe5006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.879+0000 7efe6a000700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 0x7efe5006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe6c111470 tx=0x7efe5800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.883+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7efe5c058f90 con 0x7efe6c105000 2024-04-16T07:18:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 -- 172.21.15.26:0/1827364448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1174103460 msgr2=0x7f11741038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 --2- 172.21.15.26:0/1827364448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1174103460 0x7f11741038d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1160004940 tx=0x7f1160009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 -- 172.21.15.26:0/1827364448 shutdown_connections 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 --2- 172.21.15.26:0/1827364448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1174103460 0x7f11741038d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 --2- 172.21.15.26:0/1827364448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11741094c0 0x7f1174109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 -- 172.21.15.26:0/1827364448 >> 172.21.15.26:0/1827364448 conn(0x7f1174072c30 msgr2=0x7f1174075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 -- 172.21.15.26:0/1827364448 shutdown_connections 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.843+0000 7f117b800700 1 -- 172.21.15.26:0/1827364448 wait complete. 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.844+0000 7f117b800700 1 Processor -- start 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.844+0000 7f117b800700 1 -- start start 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.844+0000 7f117b800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1174103460 0x7f1174119ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f117b800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f117b800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117411aa80 con 0x7f11741094c0 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f117b800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174114060 con 0x7f1174103460 2024-04-16T07:18:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f1173e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f1173e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39062/0 (socket says 172.21.15.26:39062) 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f1173e00700 1 -- 172.21.15.26:0/2024509758 learned_addr learned my addr 172.21.15.26:0/2024509758 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.845+0000 7f1178e00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1174103460 0x7f1174119ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1173e00700 1 -- 172.21.15.26:0/2024509758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1174103460 msgr2=0x7f1174119ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1173e00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1174103460 0x7f1174119ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1173e00700 1 -- 172.21.15.26:0/2024509758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160005040 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1173e00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1160007f50 tx=0x7f1160009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1160021070 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1160013cd0 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11741142e0 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f116000e070 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.846+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1174114840 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.848+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174104ba0 con 0x7f11741094c0 2024-04-16T07:18:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.849+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f116001c960 con 0x7f11741094c0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.849+0000 7f1170c00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 0x7f115c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.850+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f11600882f0 con 0x7f11741094c0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.850+0000 7f1178e00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 0x7f115c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.851+0000 7f1178e00700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 0x7f115c06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1168009b80 tx=0x7f116800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:56.854+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f11600583f0 con 0x7f11741094c0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.075+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f117404c710 con 0x7f11741094c0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.076+0000 7f1170c00700 1 -- 172.21.15.26:0/2024509758 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1160004490 con 0x7f11741094c0 2024-04-16T07:18:57.933 INFO:teuthology.orchestra.run.smithi026.stdout:171798691853 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 msgr2=0x7f115c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 0x7f115c06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1168009b80 tx=0x7f116800b040 comp rx=0 tx=0).stop 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 msgr2=0x7f117411a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1160007f50 tx=0x7f1160009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 shutdown_connections 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f115c0692a0 0x7f115c06b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1174103460 0x7f1174119ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 --2- 172.21.15.26:0/2024509758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11741094c0 0x7f117411a430 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 >> 172.21.15.26:0/2024509758 conn(0x7f1174072c30 msgr2=0x7f1174075010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.079+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 shutdown_connections 2024-04-16T07:18:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:57.080+0000 7f117b800700 1 -- 172.21.15.26:0/2024509758 wait complete. 2024-04-16T07:18:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:57 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2024509758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T07:18:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:57 smithi104 ceph-mon[36163]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:18:58.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.110+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efe6c111220 con 0x7efe6c105000 2024-04-16T07:18:58.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.111+0000 7efe62a00700 1 -- 172.21.15.26:0/4132210667 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efe5c058b20 con 0x7efe6c105000 2024-04-16T07:18:58.110 INFO:teuthology.orchestra.run.smithi026.stdout:115964117019 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.113+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 msgr2=0x7efe5006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.113+0000 7efe71a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 0x7efe5006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe6c111470 tx=0x7efe5800b040 comp rx=0 tx=0).stop 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 msgr2=0x7efe6c115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00ba50 tx=0x7efe5c004d10 comp rx=0 tx=0).stop 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 shutdown_connections 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efe500692a0 0x7efe5006b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe6c105000 0x7efe6c115bd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 --2- 172.21.15.26:0/4132210667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe6c1059a0 0x7efe6c116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 >> 172.21.15.26:0/4132210667 conn(0x7efe6c0fabd0 msgr2=0x7efe6c0fbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:58.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 shutdown_connections 2024-04-16T07:18:58.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.114+0000 7efe71a00700 1 -- 172.21.15.26:0/4132210667 wait complete. 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.627+0000 7f0413c00700 1 -- 172.21.15.26:0/66455156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c105010 msgr2=0x7f040c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.627+0000 7f0413c00700 1 --2- 172.21.15.26:0/66455156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c105010 0x7f040c1053e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0400004940 tx=0x7f0400009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 -- 172.21.15.26:0/66455156 shutdown_connections 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 --2- 172.21.15.26:0/66455156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c1059b0 0x7f040c0ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 --2- 172.21.15.26:0/66455156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c105010 0x7f040c1053e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 -- 172.21.15.26:0/66455156 >> 172.21.15.26:0/66455156 conn(0x7f040c0faba0 msgr2=0x7f040c0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 -- 172.21.15.26:0/66455156 shutdown_connections 2024-04-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 -- 172.21.15.26:0/66455156 wait complete. 2024-04-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.628+0000 7f0413c00700 1 Processor -- start 2024-04-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f0413c00700 1 -- start start 2024-04-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f0413c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c105010 0x7f040c115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f0413c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f0413c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1168d0 con 0x7f040c1059b0 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f0413c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c10fce0 con 0x7f040c105010 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f040be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f040be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44690/0 (socket says 172.21.15.26:44690) 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.629+0000 7f040be00700 1 -- 172.21.15.26:0/1735983414 learned_addr learned my addr 172.21.15.26:0/1735983414 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.630+0000 7f0411200700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c105010 0x7f040c115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.630+0000 7f040be00700 1 -- 172.21.15.26:0/1735983414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c105010 msgr2=0x7f040c115d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.630+0000 7f040be00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c105010 0x7f040c115d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.630+0000 7f040be00700 1 -- 172.21.15.26:0/1735983414 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400005040 con 0x7f040c1059b0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.630+0000 7f040be00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03f8002a50 tx=0x7f03f8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.631+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f8007cc0 con 0x7f040c1059b0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.631+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f800f040 con 0x7f040c1059b0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.631+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f800daf0 con 0x7f040c1059b0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.631+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c10ffc0 con 0x7f040c1059b0 2024-04-16T07:18:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.631+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c110580 con 0x7f040c1059b0 2024-04-16T07:18:58.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.632+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c100760 con 0x7f040c1059b0 2024-04-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.633+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f03f8009810 con 0x7f040c1059b0 2024-04-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.634+0000 7f0408c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 0x7f03f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.635+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03f8086be0 con 0x7f040c1059b0 2024-04-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.635+0000 7f0411200700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 0x7f03f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.635+0000 7f0411200700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 0x7f03f406b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f040000b020 tx=0x7f0400009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.639+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f03f80531b0 con 0x7f040c1059b0 2024-04-16T07:18:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4132210667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T07:18:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:58 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/4132210667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T07:18:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.861+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f040c10c110 con 0x7f040c1059b0 2024-04-16T07:18:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.861+0000 7f0408c00700 1 -- 172.21.15.26:0/1735983414 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f03f80567d0 con 0x7f040c1059b0 2024-04-16T07:18:58.860 INFO:teuthology.orchestra.run.smithi026.stdout:150323855380 2024-04-16T07:18:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.864+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 msgr2=0x7f03f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.864+0000 7f0413c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 0x7f03f406b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f040000b020 tx=0x7f0400009e30 comp rx=0 tx=0).stop 2024-04-16T07:18:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.864+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 msgr2=0x7f040c116280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:18:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.864+0000 7f0413c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03f8002a50 tx=0x7f03f8004600 comp rx=0 tx=0).stop 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 shutdown_connections 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f03f40692b0 0x7f03f406b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f040c105010 0x7f040c115d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 --2- 172.21.15.26:0/1735983414 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f040c1059b0 0x7f040c116280 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 >> 172.21.15.26:0/1735983414 conn(0x7f040c0faba0 msgr2=0x7f040c0fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 shutdown_connections 2024-04-16T07:18:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:18:58.865+0000 7f0413c00700 1 -- 172.21.15.26:0/1735983414 wait complete. 2024-04-16T07:18:58.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691853 for osd.5 2024-04-16T07:18:58.998 DEBUG:teuthology.parallel:result is None 2024-04-16T07:18:59.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542181 got 60129542183 for osd.1 2024-04-16T07:18:59.555 DEBUG:teuthology.parallel:result is None 2024-04-16T07:18:59.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:59 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1735983414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T07:18:59.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:18:59 smithi026 ceph-mon[28805]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:00.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855378 got 150323855380 for osd.4 2024-04-16T07:19:00.045 DEBUG:teuthology.parallel:result is None 2024-04-16T07:19:00.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:59 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1735983414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T07:19:00.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:18:59 smithi104 ceph-mon[36163]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:00.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117017 got 115964117019 for osd.3 2024-04-16T07:19:00.485 DEBUG:teuthology.parallel:result is None 2024-04-16T07:19:00.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-04-16T07:19:00.933 DEBUG:teuthology.parallel:result is None 2024-04-16T07:19:00.933 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-16T07:19:00.933 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T07:19:01.448 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:01.605 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:02.613 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:02 smithi026 ceph-mon[28805]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:02 smithi104 ceph-mon[36163]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.636+0000 7f958e000700 1 -- 172.21.15.26:0/314395706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9588103460 msgr2=0x7f95881038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.636+0000 7f958e000700 1 --2- 172.21.15.26:0/314395706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9588103460 0x7f95881038d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9574004940 tx=0x7f9574009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 -- 172.21.15.26:0/314395706 shutdown_connections 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 --2- 172.21.15.26:0/314395706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9588103460 0x7f95881038d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 --2- 172.21.15.26:0/314395706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95881094c0 0x7f9588109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 -- 172.21.15.26:0/314395706 >> 172.21.15.26:0/314395706 conn(0x7f9588072c30 msgr2=0x7f9588075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 -- 172.21.15.26:0/314395706 shutdown_connections 2024-04-16T07:19:03.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.637+0000 7f958e000700 1 -- 172.21.15.26:0/314395706 wait complete. 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 Processor -- start 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 -- start start 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 0x7f9588119f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958811aaf0 con 0x7f95881094c0 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.638+0000 7f958e000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9588114110 con 0x7f9588103460 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44710/0 (socket says 172.21.15.26:44710) 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 -- 172.21.15.26:0/371110347 learned_addr learned my addr 172.21.15.26:0/371110347 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9587400700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 0x7f9588119f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 -- 172.21.15.26:0/371110347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 msgr2=0x7f9588119f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 0x7f9588119f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9586a00700 1 -- 172.21.15.26:0/371110347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574005040 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.639+0000 7f9587400700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 0x7f9588119f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f9586a00700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9574009180 tx=0x7f9574009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f957400e040 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f957400bbc0 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f957401c630 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9588114390 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.640+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9588114950 con 0x7f95881094c0 2024-04-16T07:19:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.641+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588104ba0 con 0x7f95881094c0 2024-04-16T07:19:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.643+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f957400bd30 con 0x7f95881094c0 2024-04-16T07:19:03.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.644+0000 7f957f400700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 0x7f956c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:03.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.644+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9574088240 con 0x7f95881094c0 2024-04-16T07:19:03.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.644+0000 7f9587400700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 0x7f956c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:03.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.645+0000 7f9587400700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 0x7f956c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9578009fd0 tx=0x7f957800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:03.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.649+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9574054890 con 0x7f95881094c0 2024-04-16T07:19:03.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.862+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9588110530 con 0x7f956c0692a0 2024-04-16T07:19:03.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.864+0000 7f957f400700 1 -- 172.21.15.26:0/371110347 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+26907 (secure 0 0 0) 0x7f9588110530 con 0x7f956c0692a0 2024-04-16T07:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 msgr2=0x7f956c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 0x7f956c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9578009fd0 tx=0x7f957800b040 comp rx=0 tx=0).stop 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 msgr2=0x7f958811a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9574009180 tx=0x7f9574009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 shutdown_connections 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f956c0692a0 0x7f956c06b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9588103460 0x7f9588119f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 --2- 172.21.15.26:0/371110347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95881094c0 0x7f958811a4a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.867+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 >> 172.21.15.26:0/371110347 conn(0x7f9588072c30 msgr2=0x7f9588105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.868+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 shutdown_connections 2024-04-16T07:19:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:03.868+0000 7f958e000700 1 -- 172.21.15.26:0/371110347 wait complete. 2024-04-16T07:19:03.868 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T07:19:04.599 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:04 smithi026 ceph-mon[28805]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:04.650 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-16T07:19:03.306428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788080,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682832,"statfs":{"total":575970213888,"available":574139219968,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002791"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-16T07:18:03.342647+0000","last_change":"2024-04-16T07:17:29.370212+0000","last_active":"2024-04-16T07:18:03.342647+0000","last_peered":"2024-04-16T07:18:03.342647+0000","last_clean":"2024-04-16T07:18:03.342647+0000","last_became_active":"2024-04-16T07:17:29.369432+0000","last_became_peered":"2024-04-16T07:17:29.369432+0000","last_unstale":"2024-04-16T07:18:03.342647+0000","last_undegraded":"2024-04-16T07:18:03.342647+0000","last_fullsized":"2024-04-16T07:18:03.342647+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:16:23.982927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:16:23.982927+0000","last_clean_scrub_stamp":"2024-04-16T07:16:23.982927+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:03:28.169889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":4,"up_from":35,"seq":150323855381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.98899999999999999}]},{"osd":1,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.94099999999999995}]},{"osd":2,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.143}]},{"osd":3,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.026,"15min":1.03},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.274,"5min":1.381,"15min":1.381},"last":1.119},{"interface":"front","average":{"1min":1.016,"5min":1.1160000000000001,"15min":1.1319999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.452,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":1.169}]},{"osd":1,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.079,"15min":1.0389999999999999},"min":{"1min":0.70099999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":5.851,"5min":5.851,"15min":5.851},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0680000000000001,"15min":1.069},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.238}]},{"osd":2,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.135,"15min":1.127},"min":{"1min":0.78100000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.4890000000000001,"5min":1.587,"15min":1.587},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.1850000000000001,"15min":1.1899999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4970000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.4970000000000001}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.575}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.82799999999999996,"15min":0.81599999999999995},"min":{"1min":0.57799999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.90900000000000003,"15min":0.90300000000000002},"min":{"1min":0.625,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.91100000000000003}]},{"osd":1,"last update":"Tue Apr 16 07:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":0.97999999999999998},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.3740000000000001,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.052,"15min":1.0600000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.407,"5min":1.494,"15min":1.494},"last":0.98499999999999999}]},{"osd":3,"last update":"Tue Apr 16 07:18:00 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.88},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.22}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.159},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.96199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.77900000000000003,"15min":0.72599999999999998},"min":{"1min":0.625,"5min":0.438,"15min":0.438},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.008,"5min":0.83499999999999996,"15min":0.76600000000000001},"min":{"1min":0.55500000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1.046}]},{"osd":2,"last update":"Tue Apr 16 07:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":1.008},{"interface":"front","average":{"1min":1.127,"5min":1.036,"15min":1.0209999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":1.371}]},{"osd":3,"last update":"Tue Apr 16 07:18:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":1.002,"5min":1.002,"15min":1.002},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.476}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.518},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.5669999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.73999999999999999,"15min":0.71299999999999997},"min":{"1min":0.54800000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.83499999999999996,"15min":0.78000000000000003},"min":{"1min":0.52200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3620000000000001,"5min":1.734,"15min":1.734},"last":0.68200000000000005}]},{"osd":2,"last update":"Tue Apr 16 07:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.99199999999999999,"15min":0.97999999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.96399999999999997,"15min":0.94799999999999995},"min":{"1min":0.64100000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.153}]},{"osd":3,"last update":"Tue Apr 16 07:17:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.2},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":1.075}]},{"osd":4,"last update":"Tue Apr 16 07:18:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.998},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T07:19:04.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T07:19:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:04 smithi104 ceph-mon[36163]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:05.143 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:05.636 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:05 smithi026 ceph-mon[28805]: from='client.14414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T07:19:05.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:05 smithi104 ceph-mon[36163]: from='client.14414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T07:19:06.690 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:06 smithi026 ceph-mon[28805]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:06.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:06 smithi104 ceph-mon[36163]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.194+0000 7ff4b9800700 1 -- 172.21.15.26:0/3278877092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b41059b0 msgr2=0x7ff4b40ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.194+0000 7ff4b9800700 1 --2- 172.21.15.26:0/3278877092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b41059b0 0x7ff4b40ff020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff49c004940 tx=0x7ff49c009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.194+0000 7ff4b9800700 1 -- 172.21.15.26:0/3278877092 shutdown_connections 2024-04-16T07:19:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.194+0000 7ff4b9800700 1 --2- 172.21.15.26:0/3278877092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b41059b0 0x7ff4b40ff020 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.194+0000 7ff4b9800700 1 --2- 172.21.15.26:0/3278877092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b4105010 0x7ff4b41053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.195+0000 7ff4b9800700 1 -- 172.21.15.26:0/3278877092 >> 172.21.15.26:0/3278877092 conn(0x7ff4b40faba0 msgr2=0x7ff4b40fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.195+0000 7ff4b9800700 1 -- 172.21.15.26:0/3278877092 shutdown_connections 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.195+0000 7ff4b9800700 1 -- 172.21.15.26:0/3278877092 wait complete. 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.195+0000 7ff4b9800700 1 Processor -- start 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b9800700 1 -- start start 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b9800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b41059b0 0x7ff4b41162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b9800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4116900 con 0x7ff4b4105010 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b9800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b410fcd0 con 0x7ff4b41059b0 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44732/0 (socket says 172.21.15.26:44732) 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.196+0000 7ff4b2a00700 1 -- 172.21.15.26:0/1108428664 learned_addr learned my addr 172.21.15.26:0/1108428664 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:07.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4abe00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b41059b0 0x7ff4b41162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:07.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4b2a00700 1 -- 172.21.15.26:0/1108428664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b41059b0 msgr2=0x7ff4b41162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4b2a00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b41059b0 0x7ff4b41162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4b2a00700 1 -- 172.21.15.26:0/1108428664 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c005040 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4b2a00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4002a50 tx=0x7ff4a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.197+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a4007cc0 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.198+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4a400f040 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.198+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a4009640 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.198+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b410ffb0 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.198+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b4110570 con 0x7ff4b4105010 2024-04-16T07:19:07.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.199+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b4100760 con 0x7ff4b4105010 2024-04-16T07:19:07.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.200+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff4a40097a0 con 0x7ff4b4105010 2024-04-16T07:19:07.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.201+0000 7ff4aaa00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 0x7ff49806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:07.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.201+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff4a4086900 con 0x7ff4b4105010 2024-04-16T07:19:07.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.201+0000 7ff4abe00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 0x7ff49806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:07.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.202+0000 7ff4abe00700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 0x7ff49806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4111610 tx=0x7ff49c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:07.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.206+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff4a4056a00 con 0x7ff4b4105010 2024-04-16T07:19:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.416+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4b404bf40 con 0x7ff4980692b0 2024-04-16T07:19:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.418+0000 7ff4aaa00700 1 -- 172.21.15.26:0/1108428664 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27862 (secure 0 0 0) 0x7ff4b404bf40 con 0x7ff4980692b0 2024-04-16T07:19:07.417 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.420+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 msgr2=0x7ff49806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.420+0000 7ff4b9800700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 0x7ff49806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4111610 tx=0x7ff49c009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 msgr2=0x7ff4b4115d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4002a50 tx=0x7ff4a40046b0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 shutdown_connections 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff4980692b0 0x7ff49806b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4b4105010 0x7ff4b4115d70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 --2- 172.21.15.26:0/1108428664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4b41059b0 0x7ff4b41162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 >> 172.21.15.26:0/1108428664 conn(0x7ff4b40faba0 msgr2=0x7ff4b40fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 shutdown_connections 2024-04-16T07:19:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:07.421+0000 7ff4b9800700 1 -- 172.21.15.26:0/1108428664 wait complete. 2024-04-16T07:19:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T07:19:08.300 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-16T07:19:07.307645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788080,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682832,"statfs":{"total":575970213888,"available":574139219968,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003245"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-16T07:18:03.342647+0000","last_change":"2024-04-16T07:17:29.370212+0000","last_active":"2024-04-16T07:18:03.342647+0000","last_peered":"2024-04-16T07:18:03.342647+0000","last_clean":"2024-04-16T07:18:03.342647+0000","last_became_active":"2024-04-16T07:17:29.369432+0000","last_became_peered":"2024-04-16T07:17:29.369432+0000","last_unstale":"2024-04-16T07:18:03.342647+0000","last_undegraded":"2024-04-16T07:18:03.342647+0000","last_fullsized":"2024-04-16T07:18:03.342647+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:16:23.982927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:16:23.982927+0000","last_clean_scrub_stamp":"2024-04-16T07:16:23.982927+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T16:03:28.169889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.008}]},{"osd":1,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":1.25}]},{"osd":2,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.194},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.335}]},{"osd":3,"last update":"Tue Apr 16 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.026,"15min":1.03},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.274,"5min":1.381,"15min":1.381},"last":1.103},{"interface":"front","average":{"1min":1.016,"5min":1.1160000000000001,"15min":1.1319999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.452,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.92800000000000005}]},{"osd":1,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.079,"15min":1.0389999999999999},"min":{"1min":0.70099999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":5.851,"5min":5.851,"15min":5.851},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0680000000000001,"15min":1.069},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.2090000000000001}]},{"osd":2,"last update":"Tue Apr 16 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.135,"15min":1.127},"min":{"1min":0.78100000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.4890000000000001,"5min":1.587,"15min":1.587},"last":1.258},{"interface":"front","average":{"1min":1.155,"5min":1.1850000000000001,"15min":1.1899999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4970000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.4359999999999999}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.292}]},{"osd":5,"last update":"Tue Apr 16 07:19:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.383},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":1.129,"5min":1.129,"15min":1.129},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.349}]}]},{"osd":2,"up_from":19,"seq":81604378659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.82799999999999996,"15min":0.81599999999999995},"min":{"1min":0.57799999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.90900000000000003,"15min":0.90300000000000002},"min":{"1min":0.625,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.91100000000000003}]},{"osd":1,"last update":"Tue Apr 16 07:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":0.97999999999999998},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.3740000000000001,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.052,"15min":1.0600000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.407,"5min":1.494,"15min":1.494},"last":0.98499999999999999}]},{"osd":3,"last update":"Tue Apr 16 07:18:00 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.88},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.22}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.159},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.96199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.77900000000000003,"15min":0.72599999999999998},"min":{"1min":0.625,"5min":0.438,"15min":0.438},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.008,"5min":0.83499999999999996,"15min":0.76600000000000001},"min":{"1min":0.55500000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1.1439999999999999}]},{"osd":2,"last update":"Tue Apr 16 07:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.036,"15min":1.0209999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":1.0149999999999999}]},{"osd":3,"last update":"Tue Apr 16 07:19:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.181,"15min":1.171},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.218},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.296,"15min":1.296},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.4690000000000001}]},{"osd":4,"last update":"Tue Apr 16 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.4139999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.73999999999999999,"15min":0.71299999999999997},"min":{"1min":0.54800000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.83499999999999996,"15min":0.78000000000000003},"min":{"1min":0.52200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3620000000000001,"5min":1.734,"15min":1.734},"last":0.84199999999999997}]},{"osd":2,"last update":"Tue Apr 16 07:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.99199999999999999,"15min":0.97999999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.96399999999999997,"15min":0.94799999999999995},"min":{"1min":0.64100000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.98199999999999998}]},{"osd":3,"last update":"Tue Apr 16 07:19:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.137,"15min":1.1339999999999999},"min":{"1min":0.91300000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.411,"5min":1.488,"15min":1.488},"last":1.2669999999999999}]},{"osd":4,"last update":"Tue Apr 16 07:18:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.145}]},{"osd":5,"last update":"Tue Apr 16 07:19:04 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.319},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.2330000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T07:19:08.302 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-16T07:19:08.302 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:19:08.302 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:19:08.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-16T07:19:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:08 smithi026 ceph-mon[28805]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:08 smithi026 ceph-mon[28805]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T07:19:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:08 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:08 smithi104 ceph-mon[36163]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:08 smithi104 ceph-mon[36163]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T07:19:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:08 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:09.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:09.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:09.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:09.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:09.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:09.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:10.622 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:10 smithi026 ceph-mon[28805]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:10.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 -- 172.21.15.26:0/4076341818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 msgr2=0x7ff008103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 --2- 172.21.15.26:0/4076341818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008103010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feffc004940 tx=0x7feffc009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 -- 172.21.15.26:0/4076341818 shutdown_connections 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 --2- 172.21.15.26:0/4076341818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 0x7ff0080fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 --2- 172.21.15.26:0/4076341818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008103010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.787+0000 7ff00f000700 1 -- 172.21.15.26:0/4076341818 >> 172.21.15.26:0/4076341818 conn(0x7ff0080f8790 msgr2=0x7ff0080faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.788+0000 7ff00f000700 1 -- 172.21.15.26:0/4076341818 shutdown_connections 2024-04-16T07:19:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.788+0000 7ff00f000700 1 -- 172.21.15.26:0/4076341818 wait complete. 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 Processor -- start 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 -- start start 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 0x7ff008116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0081167a0 con 0x7ff008102c40 2024-04-16T07:19:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.789+0000 7ff00f000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00810fb90 con 0x7ff0081035e0 2024-04-16T07:19:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49784/0 (socket says 172.21.15.26:49784) 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 -- 172.21.15.26:0/1127982511 learned_addr learned my addr 172.21.15.26:0/1127982511 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00d200700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 0x7ff008116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 -- 172.21.15.26:0/1127982511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 msgr2=0x7ff008116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 0x7ff008116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 -- 172.21.15.26:0/1127982511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc005040 con 0x7ff008102c40 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.790+0000 7ff00dc00700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feffc009150 tx=0x7feffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.791+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feffc00e040 con 0x7ff008102c40 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.791+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feffc013420 con 0x7ff008102c40 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.791+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00810fe10 con 0x7ff008102c40 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.791+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feffc01c630 con 0x7ff008102c40 2024-04-16T07:19:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.791+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0081103d0 con 0x7ff008102c40 2024-04-16T07:19:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.792+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0080fe390 con 0x7ff008102c40 2024-04-16T07:19:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.793+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7feffc013590 con 0x7ff008102c40 2024-04-16T07:19:10.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.794+0000 7ff005600700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 0x7feff006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:10.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.794+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feffc087ff0 con 0x7ff008102c40 2024-04-16T07:19:10.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.794+0000 7ff00d200700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 0x7feff006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:10.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.795+0000 7ff00d200700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 0x7feff006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff008111440 tx=0x7feff4009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:10.799+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7feffc058190 con 0x7ff008102c40 2024-04-16T07:19:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:10 smithi104 ceph-mon[36163]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:11.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.064+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff00810ffa0 con 0x7ff008102c40 2024-04-16T07:19:11.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.065+0000 7ff005600700 1 -- 172.21.15.26:0/1127982511 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff00810ffa0 con 0x7ff008102c40 2024-04-16T07:19:11.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:11.063 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:19:11.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.067+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 msgr2=0x7feff006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.067+0000 7ff00f000700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 0x7feff006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff008111440 tx=0x7feff4009ee0 comp rx=0 tx=0).stop 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.067+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 msgr2=0x7ff008115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.067+0000 7ff00f000700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feffc009150 tx=0x7feffc009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 shutdown_connections 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7feff00692a0 0x7feff006b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff008102c40 0x7ff008115c10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 --2- 172.21.15.26:0/1127982511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0081035e0 0x7ff008116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 >> 172.21.15.26:0/1127982511 conn(0x7ff0080f8790 msgr2=0x7ff0080f96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 shutdown_connections 2024-04-16T07:19:11.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:11.068+0000 7ff00f000700 1 -- 172.21.15.26:0/1127982511 wait complete. 2024-04-16T07:19:11.715 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:11 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1127982511' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T07:19:11.807 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:19:11.807 INFO:tasks.cephadm:Setup complete, yielding 2024-04-16T07:19:11.807 INFO:teuthology.run_tasks:Running task print... 2024-04-16T07:19:11.817 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-16T07:19:11.817 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:11.840 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:11.840 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-16T07:19:11.841 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:11 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1127982511' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T07:19:12.339 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:12.674 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:12 smithi026 ceph-mon[28805]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:12 smithi104 ceph-mon[36163]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:14.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:14 smithi026 ceph-mon[28805]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:14.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3303556969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 msgr2=0x7f1ee40ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:14.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3303556969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee40ff800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8004940 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3303556969 shutdown_connections 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3303556969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 0x7f1ee4102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3303556969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee40ff800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3303556969 >> 172.21.15.26:0/3303556969 conn(0x7f1ee40f7270 msgr2=0x7f1ee40f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3303556969 shutdown_connections 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.540+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3303556969 wait complete. 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.541+0000 7f1eeaa00700 1 Processor -- start 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1eeaa00700 1 -- start start 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1eeaa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1eeaa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 0x7f1ee410dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1eeaa00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee410e3e0 con 0x7f1ee40662f0 2024-04-16T07:19:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1eeaa00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee410ebe0 con 0x7f1ee40ffd40 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1ee9600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1ee9600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49808/0 (socket says 172.21.15.26:49808) 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.542+0000 7f1ee9600700 1 -- 172.21.15.26:0/3520227904 learned_addr learned my addr 172.21.15.26:0/3520227904 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.543+0000 7f1ee8c00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 0x7f1ee410dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.543+0000 7f1ee9600700 1 -- 172.21.15.26:0/3520227904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 msgr2=0x7f1ee410dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.543+0000 7f1ee9600700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 0x7f1ee410dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.543+0000 7f1ee9600700 1 -- 172.21.15.26:0/3520227904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8005040 con 0x7f1ee40662f0 2024-04-16T07:19:14.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.543+0000 7f1ee9600700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800b0d0 tx=0x7f1ed8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.544+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed800e070 con 0x7f1ee40662f0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.544+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ed800b410 con 0x7f1ee40662f0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.544+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee410ee60 con 0x7f1ee40662f0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.544+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed8012600 con 0x7f1ee40662f0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.544+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee410f3a0 con 0x7f1ee40662f0 2024-04-16T07:19:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.545+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee405f5f0 con 0x7f1ee40662f0 2024-04-16T07:19:14.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.546+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1ed800ba60 con 0x7f1ee40662f0 2024-04-16T07:19:14.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.547+0000 7f1ee1600700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 0x7f1ecc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:14.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.547+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ed8087d10 con 0x7f1ee40662f0 2024-04-16T07:19:14.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.547+0000 7f1ee8c00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 0x7f1ecc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:14.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.548+0000 7f1ee8c00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 0x7f1ecc06b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0004600 tx=0x7f1ed000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:14.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.552+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1ed8057d70 con 0x7f1ee40662f0 2024-04-16T07:19:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.768+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f1ee404bed0 con 0x7f1ee40662f0 2024-04-16T07:19:14.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.779+0000 7f1ee1600700 1 -- 172.21.15.26:0/3520227904 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f1ed80220d0 con 0x7f1ee40662f0 2024-04-16T07:19:14.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 msgr2=0x7f1ecc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:14.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 0x7f1ecc06b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0004600 tx=0x7f1ed000b040 comp rx=0 tx=0).stop 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 msgr2=0x7f1ee410d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800b0d0 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 shutdown_connections 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f1ecc0692b0 0x7f1ecc06b760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40662f0 0x7f1ee410d8a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 --2- 172.21.15.26:0/3520227904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ee40ffd40 0x7f1ee410dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:14.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 >> 172.21.15.26:0/3520227904 conn(0x7f1ee40f7270 msgr2=0x7f1ee40f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:14.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 shutdown_connections 2024-04-16T07:19:14.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:14.782+0000 7f1eeaa00700 1 -- 172.21.15.26:0/3520227904 wait complete. 2024-04-16T07:19:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:14 smithi104 ceph-mon[36163]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:15.672 INFO:teuthology.run_tasks:Running task print... 2024-04-16T07:19:15.681 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-16T07:19:15.681 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:15.698 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:15.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-16T07:19:16.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3520227904' entity='client.admin' 2024-04-16T07:19:16.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:16.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:16.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:16.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:16.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:15 smithi026 ceph-mon[28805]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:16.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3520227904' entity='client.admin' 2024-04-16T07:19:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:15 smithi104 ceph-mon[36163]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:16.210 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:17.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:17.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:18.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:17 smithi104 ceph-mon[36163]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:18.129 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:17 smithi026 ceph-mon[28805]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:18.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 -- 172.21.15.26:0/4034985893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 msgr2=0x7f85640ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:18.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 --2- 172.21.15.26:0/4034985893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f85640ff400 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8554004940 tx=0x7f8554009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:18.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 -- 172.21.15.26:0/4034985893 shutdown_connections 2024-04-16T07:19:18.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 --2- 172.21.15.26:0/4034985893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 0x7f8564100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 --2- 172.21.15.26:0/4034985893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f85640ff400 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 -- 172.21.15.26:0/4034985893 >> 172.21.15.26:0/4034985893 conn(0x7f85640fa950 msgr2=0x7f85640fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:18.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 -- 172.21.15.26:0/4034985893 shutdown_connections 2024-04-16T07:19:18.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.304+0000 7f8568c00700 1 -- 172.21.15.26:0/4034985893 wait complete. 2024-04-16T07:19:18.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.305+0000 7f8568c00700 1 Processor -- start 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.305+0000 7f8568c00700 1 -- start start 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8568c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8568c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 0x7f856410fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8568c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85641105f0 con 0x7f85640feff0 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8568c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8564110760 con 0x7f85641002a0 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8561e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8561e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56044/0 (socket says 172.21.15.26:56044) 2024-04-16T07:19:18.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8561e00700 1 -- 172.21.15.26:0/3360331144 learned_addr learned my addr 172.21.15.26:0/3360331144 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:18.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.306+0000 7f8561400700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 0x7f856410fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:18.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f8561e00700 1 -- 172.21.15.26:0/3360331144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 msgr2=0x7f856410fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:18.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f8561e00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 0x7f856410fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f8561e00700 1 -- 172.21.15.26:0/3360331144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554005040 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f8561e00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8554013870 tx=0x7f8554009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f855400e070 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f855401c400 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8564115160 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.307+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f855400b630 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.308+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8564115640 con 0x7f85640feff0 2024-04-16T07:19:18.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.309+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856404c710 con 0x7f85640feff0 2024-04-16T07:19:18.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.310+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f855401c570 con 0x7f85640feff0 2024-04-16T07:19:18.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.311+0000 7f855a000700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:18.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.311+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8554088510 con 0x7f85640feff0 2024-04-16T07:19:18.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.311+0000 7f8561400700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:18.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.312+0000 7f8561400700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 0x7f854806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8550004600 tx=0x7f855000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:18.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.316+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f85540586b0 con 0x7f85640feff0 2024-04-16T07:19:18.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.546+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8564104540 con 0x7f85480692a0 2024-04-16T07:19:18.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.548+0000 7f855a000700 1 -- 172.21.15.26:0/3360331144 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8564104540 con 0x7f85480692a0 2024-04-16T07:19:18.547 INFO:teuthology.orchestra.run.smithi026.stdout:Backend: cephadm 2024-04-16T07:19:18.547 INFO:teuthology.orchestra.run.smithi026.stdout:Available: Yes 2024-04-16T07:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout:Paused: No 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 msgr2=0x7f854806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 0x7f854806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8550004600 tx=0x7f855000b040 comp rx=0 tx=0).stop 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 msgr2=0x7f856410fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8554013870 tx=0x7f8554009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 shutdown_connections 2024-04-16T07:19:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f85480692a0 0x7f854806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85640feff0 0x7f856410fab0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 --2- 172.21.15.26:0/3360331144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85641002a0 0x7f856410fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 >> 172.21.15.26:0/3360331144 conn(0x7f85640fa950 msgr2=0x7f85640fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 shutdown_connections 2024-04-16T07:19:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:18.551+0000 7f8568c00700 1 -- 172.21.15.26:0/3360331144 wait complete. 2024-04-16T07:19:19.258 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:18 smithi026 ceph-mon[28805]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:19.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:18 smithi104 ceph-mon[36163]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:19.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-16T07:19:19.977 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:20.133 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:19 smithi026 ceph-mon[28805]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:20.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:19 smithi104 ceph-mon[36163]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 -- 172.21.15.26:0/2200615021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f841046d0 msgr2=0x7f6f84104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 --2- 172.21.15.26:0/2200615021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f841046d0 0x7f6f84104b40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f70004940 tx=0x7f6f70009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 -- 172.21.15.26:0/2200615021 shutdown_connections 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 --2- 172.21.15.26:0/2200615021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f841046d0 0x7f6f84104b40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 --2- 172.21.15.26:0/2200615021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f84103420 0x7f6f84103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.053+0000 7f6f8c000700 1 -- 172.21.15.26:0/2200615021 >> 172.21.15.26:0/2200615021 conn(0x7f6f84072640 msgr2=0x7f6f84074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.054+0000 7f6f8c000700 1 -- 172.21.15.26:0/2200615021 shutdown_connections 2024-04-16T07:19:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.054+0000 7f6f8c000700 1 -- 172.21.15.26:0/2200615021 wait complete. 2024-04-16T07:19:22.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.054+0000 7f6f8c000700 1 Processor -- start 2024-04-16T07:19:22.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f8c000700 1 -- start start 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f8c000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f8c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f841046d0 0x7f6f84114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f8c000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84114a90 con 0x7f6f84103420 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f8c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84114c00 con 0x7f6f841046d0 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f89600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f89600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56064/0 (socket says 172.21.15.26:56064) 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.055+0000 7f6f89600700 1 -- 172.21.15.26:0/3561121173 learned_addr learned my addr 172.21.15.26:0/3561121173 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f88c00700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f841046d0 0x7f6f84114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f89600700 1 -- 172.21.15.26:0/3561121173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f841046d0 msgr2=0x7f6f84114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f89600700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f841046d0 0x7f6f84114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f89600700 1 -- 172.21.15.26:0/3561121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70005040 con 0x7f6f84103420 2024-04-16T07:19:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f89600700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f78002a50 tx=0x7f6f780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f78007cc0 con 0x7f6f84103420 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f7800f040 con 0x7f6f84103420 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84119660 con 0x7f6f84103420 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.056+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f78009640 con 0x7f6f84103420 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.057+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f84119c20 con 0x7f6f84103420 2024-04-16T07:19:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.058+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8404c710 con 0x7f6f84103420 2024-04-16T07:19:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.059+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6f780097a0 con 0x7f6f84103420 2024-04-16T07:19:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.060+0000 7f6f81600700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 0x7f6f6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.060+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6f78087310 con 0x7f6f84103420 2024-04-16T07:19:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.060+0000 7f6f88c00700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 0x7f6f6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.061+0000 7f6f88c00700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 0x7f6f6c06b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6f70004f90 tx=0x7f6f70009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:22.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.064+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6f78057370 con 0x7f6f84103420 2024-04-16T07:19:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.292+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f84108a10 con 0x7f6f6c0692a0 2024-04-16T07:19:22.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.302+0000 7f6f81600700 1 -- 172.21.15.26:0/3561121173 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f6f84108a10 con 0x7f6f6c0692a0 2024-04-16T07:19:22.301 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (4m) 2m ago 6m 13.6M - 0.25.0 c8568f914cd2 d5f7c216b68d 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (6m) 2m ago 6m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (5m) 79s ago 5m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (4m) 2m ago 6m 74.2M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (8m) 2m ago 8m 473M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (5m) 79s ago 5m 431M - 17.2.7-847-g45317f95 b333581ccdf5 275c78dfc5a9 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (8m) 2m ago 8m 41.0M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (5m) 79s ago 5m 37.5M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (6m) 2m ago 6m 7298k - 1.5.0 0da6a335fe13 5198001b496e 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 79s ago 5m 7666k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (4m) 2m ago 4m 36.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (3m) 2m ago 3m 39.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (3m) 2m ago 3m 37.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:19:22.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (2m) 79s ago 2m 38.3M 5312M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:19:22.303 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (119s) 79s ago 119s 36.6M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:19:22.303 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (85s) 79s ago 85s 10.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:19:22.303 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (4m) 2m ago 6m 37.2M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:19:22.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 msgr2=0x7f6f6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 0x7f6f6c06b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6f70004f90 tx=0x7f6f70009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 msgr2=0x7f6f84113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f78002a50 tx=0x7f6f780046b0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 shutdown_connections 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6f6c0692a0 0x7f6f6c06b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f84103420 0x7f6f84113f50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.306+0000 7f6f8c000700 1 --2- 172.21.15.26:0/3561121173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f841046d0 0x7f6f84114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.307+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 >> 172.21.15.26:0/3561121173 conn(0x7f6f84072640 msgr2=0x7f6f84073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.307+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 shutdown_connections 2024-04-16T07:19:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:22.307+0000 7f6f8c000700 1 -- 172.21.15.26:0/3561121173 wait complete. 2024-04-16T07:19:22.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:22 smithi026 ceph-mon[28805]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:22 smithi104 ceph-mon[36163]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:23.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-16T07:19:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:23.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:23 smithi026 ceph-mon[28805]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:23.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:23 smithi104 ceph-mon[36163]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:24.526 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:24 smithi026 ceph-mon[28805]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:24 smithi104 ceph-mon[36163]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.683+0000 7f43ce800700 1 -- 172.21.15.26:0/434984321 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c81046d0 msgr2=0x7f43c8104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.683+0000 7f43ce800700 1 --2- 172.21.15.26:0/434984321 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c81046d0 0x7f43c8104b40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004940 tx=0x7f43b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 -- 172.21.15.26:0/434984321 shutdown_connections 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 --2- 172.21.15.26:0/434984321 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c81046d0 0x7f43c8104b40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 --2- 172.21.15.26:0/434984321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c8103420 0x7f43c8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 -- 172.21.15.26:0/434984321 >> 172.21.15.26:0/434984321 conn(0x7f43c8072640 msgr2=0x7f43c8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 -- 172.21.15.26:0/434984321 shutdown_connections 2024-04-16T07:19:25.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 -- 172.21.15.26:0/434984321 wait complete. 2024-04-16T07:19:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.684+0000 7f43ce800700 1 Processor -- start 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43ce800700 1 -- start start 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43ce800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43ce800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c81046d0 0x7f43c8114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43ce800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8114a90 con 0x7f43c8103420 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43ce800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8114c00 con 0x7f43c81046d0 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43c7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56088/0 (socket says 172.21.15.26:56088) 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.685+0000 7f43c7400700 1 -- 172.21.15.26:0/2664954511 learned_addr learned my addr 172.21.15.26:0/2664954511 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43c6a00700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c81046d0 0x7f43c8114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43c7400700 1 -- 172.21.15.26:0/2664954511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c81046d0 msgr2=0x7f43c8114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43c7400700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c81046d0 0x7f43c8114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43c7400700 1 -- 172.21.15.26:0/2664954511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b4005040 con 0x7f43c8103420 2024-04-16T07:19:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43c7400700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f43b8002a50 tx=0x7f43b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43b8007cc0 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43b800f040 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.686+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43b8009640 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.687+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c8119660 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.687+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c8119c20 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.688+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c804c710 con 0x7f43c8103420 2024-04-16T07:19:25.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.689+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f43b80097c0 con 0x7f43c8103420 2024-04-16T07:19:25.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.690+0000 7f43bf400700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 0x7f43ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:25.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.690+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f43b8086c60 con 0x7f43c8103420 2024-04-16T07:19:25.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.690+0000 7f43c6a00700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 0x7f43ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:25.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.691+0000 7f43c6a00700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 0x7f43ac06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004f90 tx=0x7f43b4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:25.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.695+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f43b8056cc0 con 0x7f43c8103420 2024-04-16T07:19:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.921+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f43c8108a10 con 0x7f43ac0692a0 2024-04-16T07:19:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.933+0000 7f43bf400700 1 -- 172.21.15.26:0/2664954511 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f43c8108a10 con 0x7f43ac0692a0 2024-04-16T07:19:25.932 INFO:teuthology.orchestra.run.smithi026.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:crash 2/2 2m ago 7m * 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:mgr 2/2 2m ago 7m count:2 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:mon 2/2 2m ago 6m smithi026:172.21.15.26=smithi026;smithi104:172.21.15.104=smithi104;count:2 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:osd 6 2m ago - 2024-04-16T07:19:25.933 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus ?:9095 1/1 2m ago 7m count:1 2024-04-16T07:19:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.936+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 msgr2=0x7f43ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.936+0000 7f43ce800700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 0x7f43ac06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004f90 tx=0x7f43b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.936+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 msgr2=0x7f43c8113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.936+0000 7f43ce800700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f43b8002a50 tx=0x7f43b80046b0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 shutdown_connections 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f43ac0692a0 0x7f43ac06b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c8103420 0x7f43c8113f50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 --2- 172.21.15.26:0/2664954511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c81046d0 0x7f43c8114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 >> 172.21.15.26:0/2664954511 conn(0x7f43c8072640 msgr2=0x7f43c8073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 shutdown_connections 2024-04-16T07:19:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:25.937+0000 7f43ce800700 1 -- 172.21.15.26:0/2664954511 wait complete. 2024-04-16T07:19:26.727 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:26 smithi026 ceph-mon[28805]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:26.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-16T07:19:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:26 smithi104 ceph-mon[36163]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:27.336 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:27.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:27 smithi026 ceph-mon[28805]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:27.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:27 smithi104 ceph-mon[36163]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:28.582 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:28 smithi026 ceph-mon[28805]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:28 smithi104 ceph-mon[36163]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 -- 172.21.15.26:0/3914598915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae0100250 msgr2=0x7f6ae01006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/3914598915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae0100250 0x7f6ae01006c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6acc004940 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 -- 172.21.15.26:0/3914598915 shutdown_connections 2024-04-16T07:19:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/3914598915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae0100250 0x7f6ae01006c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/3914598915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae00fefa0 0x7f6ae00ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 -- 172.21.15.26:0/3914598915 >> 172.21.15.26:0/3914598915 conn(0x7f6ae00fa790 msgr2=0x7f6ae00fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 -- 172.21.15.26:0/3914598915 shutdown_connections 2024-04-16T07:19:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.341+0000 7f6ae7e00700 1 -- 172.21.15.26:0/3914598915 wait complete. 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.342+0000 7f6ae7e00700 1 Processor -- start 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.342+0000 7f6ae7e00700 1 -- start start 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.342+0000 7f6ae7e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae7e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae0100250 0x7f6ae01100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae7e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01106e0 con 0x7f6ae00fefa0 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae7e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0110f30 con 0x7f6ae0100250 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54690/0 (socket says 172.21.15.26:54690) 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 -- 172.21.15.26:0/833352316 learned_addr learned my addr 172.21.15.26:0/833352316 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae4a00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae0100250 0x7f6ae01100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 -- 172.21.15.26:0/833352316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae0100250 msgr2=0x7f6ae01100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae0100250 0x7f6ae01100e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.343+0000 7f6ae5400700 1 -- 172.21.15.26:0/833352316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc005040 con 0x7f6ae00fefa0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6ae5400700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c830 tx=0x7f6ad4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad400a760 con 0x7f6ae00fefa0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae0111210 con 0x7f6ae00fefa0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ad400ad90 con 0x7f6ae00fefa0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad40113a0 con 0x7f6ae00fefa0 2024-04-16T07:19:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.344+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae0111750 con 0x7f6ae00fefa0 2024-04-16T07:19:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.345+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae00fe3a0 con 0x7f6ae00fefa0 2024-04-16T07:19:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.346+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6ad4012400 con 0x7f6ae00fefa0 2024-04-16T07:19:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.347+0000 7f6add600700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 0x7f6ac806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.347+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ad4086ed0 con 0x7f6ae00fefa0 2024-04-16T07:19:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.347+0000 7f6ae4a00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 0x7f6ac806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.348+0000 7f6ae4a00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 0x7f6ac806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00b3b0 tx=0x7f6acc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.351+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6ad4056fb0 con 0x7f6ae00fefa0 2024-04-16T07:19:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.572+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ae0111b80 con 0x7f6ac80692a0 2024-04-16T07:19:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.576+0000 7f6add600700 1 -- 172.21.15.26:0/833352316 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f6ae0111b80 con 0x7f6ac80692a0 2024-04-16T07:19:29.575 INFO:teuthology.orchestra.run.smithi026.stdout:HOST ADDR LABELS STATUS 2024-04-16T07:19:29.575 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 172.21.15.26 2024-04-16T07:19:29.575 INFO:teuthology.orchestra.run.smithi026.stdout:smithi104 172.21.15.104 2024-04-16T07:19:29.575 INFO:teuthology.orchestra.run.smithi026.stdout:2 hosts in cluster 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.578+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 msgr2=0x7f6ac806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.578+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 0x7f6ac806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00b3b0 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.578+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 msgr2=0x7f6ae010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.578+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c830 tx=0x7f6ad4004470 comp rx=0 tx=0).stop 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 shutdown_connections 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6ac80692a0 0x7f6ac806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6ae00fefa0 0x7f6ae010fba0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 --2- 172.21.15.26:0/833352316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ae0100250 0x7f6ae01100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 >> 172.21.15.26:0/833352316 conn(0x7f6ae00fa790 msgr2=0x7f6ae00fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 shutdown_connections 2024-04-16T07:19:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:29.579+0000 7f6ae7e00700 1 -- 172.21.15.26:0/833352316 wait complete. 2024-04-16T07:19:30.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-16T07:19:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:30 smithi026 ceph-mon[28805]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:30 smithi026 ceph-mon[28805]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:30 smithi104 ceph-mon[36163]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:30 smithi104 ceph-mon[36163]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:30.908 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:32.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:32 smithi026 ceph-mon[28805]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:32 smithi104 ceph-mon[36163]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:33.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 -- 172.21.15.26:0/2775881159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c0708f0 msgr2=0x7f678c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:33.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 --2- 172.21.15.26:0/2775881159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c0708f0 0x7f678c070d60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6778004940 tx=0x7f6778009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:33.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 -- 172.21.15.26:0/2775881159 shutdown_connections 2024-04-16T07:19:33.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 --2- 172.21.15.26:0/2775881159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c0708f0 0x7f678c070d60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 --2- 172.21.15.26:0/2775881159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c071ea0 0x7f678c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 -- 172.21.15.26:0/2775881159 >> 172.21.15.26:0/2775881159 conn(0x7f678c0f83a0 msgr2=0x7f678c0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:33.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 -- 172.21.15.26:0/2775881159 shutdown_connections 2024-04-16T07:19:33.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.045+0000 7f6791a00700 1 -- 172.21.15.26:0/2775881159 wait complete. 2024-04-16T07:19:33.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.046+0000 7f6791a00700 1 Processor -- start 2024-04-16T07:19:33.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.046+0000 7f6791a00700 1 -- start start 2024-04-16T07:19:33.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f6791a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c0708f0 0x7f678c10fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:33.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f6791a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f6791a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c110580 con 0x7f678c071ea0 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f6791a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c110dd0 con 0x7f678c0708f0 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f678b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f678b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54712/0 (socket says 172.21.15.26:54712) 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f678b400700 1 -- 172.21.15.26:0/546537377 learned_addr learned my addr 172.21.15.26:0/546537377 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.047+0000 7f678be00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c0708f0 0x7f678c10fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f678b400700 1 -- 172.21.15.26:0/546537377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c0708f0 msgr2=0x7f678c10fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f678b400700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c0708f0 0x7f678c10fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f678b400700 1 -- 172.21.15.26:0/546537377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778005040 con 0x7f678c071ea0 2024-04-16T07:19:33.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f678b400700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6778009150 tx=0x7f6778009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f677800e040 con 0x7f678c071ea0 2024-04-16T07:19:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f677800bbf0 con 0x7f678c071ea0 2024-04-16T07:19:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c111050 con 0x7f678c071ea0 2024-04-16T07:19:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f677801c630 con 0x7f678c071ea0 2024-04-16T07:19:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.048+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c111590 con 0x7f678c071ea0 2024-04-16T07:19:33.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.049+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c0fc020 con 0x7f678c071ea0 2024-04-16T07:19:33.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.050+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f677800bd60 con 0x7f678c071ea0 2024-04-16T07:19:33.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.051+0000 7f6783e00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 0x7f677006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:33.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.052+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6778088250 con 0x7f678c071ea0 2024-04-16T07:19:33.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.052+0000 7f678be00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 0x7f677006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:33.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.052+0000 7f678be00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 0x7f677006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f678c071be0 tx=0x7f677c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:33.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.057+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6778058330 con 0x7f678c071ea0 2024-04-16T07:19:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.280+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f678c111980 con 0x7f67700692a0 2024-04-16T07:19:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.282+0000 7f6783e00700 1 -- 172.21.15.26:0/546537377 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f678c111980 con 0x7f67700692a0 2024-04-16T07:19:33.281 INFO:teuthology.orchestra.run.smithi026.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-16T07:19:33.281 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760020480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-16T07:19:33.281 INFO:teuthology.orchestra.run.smithi026.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 85s ago Has a FileSystem, LVM detected 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 msgr2=0x7f677006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 0x7f677006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f678c071be0 tx=0x7f677c00b040 comp rx=0 tx=0).stop 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 msgr2=0x7f678c10ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6778009150 tx=0x7f6778009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 shutdown_connections 2024-04-16T07:19:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f67700692a0 0x7f677006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f678c0708f0 0x7f678c10fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 --2- 172.21.15.26:0/546537377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f678c071ea0 0x7f678c10ff80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:33.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 >> 172.21.15.26:0/546537377 conn(0x7f678c0f83a0 msgr2=0x7f678c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:33.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 shutdown_connections 2024-04-16T07:19:33.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:33.285+0000 7f6791a00700 1 -- 172.21.15.26:0/546537377 wait complete. 2024-04-16T07:19:34.028 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:34.042 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:34.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-16T07:19:34.555 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:34.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:34 smithi026 ceph-mon[28805]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:34.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:34 smithi026 ceph-mon[28805]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:34 smithi104 ceph-mon[36163]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:34 smithi104 ceph-mon[36163]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 -- 172.21.15.26:0/1786194779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40fff70 msgr2=0x7f89e4102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 --2- 172.21.15.26:0/1786194779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40fff70 0x7f89e4102390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f89d0004940 tx=0x7f89d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 -- 172.21.15.26:0/1786194779 shutdown_connections 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 --2- 172.21.15.26:0/1786194779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40fff70 0x7f89e4102390 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 --2- 172.21.15.26:0/1786194779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40662f0 0x7f89e40ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 -- 172.21.15.26:0/1786194779 >> 172.21.15.26:0/1786194779 conn(0x7f89e40f74c0 msgr2=0x7f89e40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 -- 172.21.15.26:0/1786194779 shutdown_connections 2024-04-16T07:19:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.605+0000 7f89ebe00700 1 -- 172.21.15.26:0/1786194779 wait complete. 2024-04-16T07:19:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.606+0000 7f89ebe00700 1 Processor -- start 2024-04-16T07:19:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.606+0000 7f89ebe00700 1 -- start start 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89ebe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89ebe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40fff70 0x7f89e40fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89ebe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40fbee0 con 0x7f89e40662f0 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89ebe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40fc050 con 0x7f89e40fff70 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54720/0 (socket says 172.21.15.26:54720) 2024-04-16T07:19:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 -- 172.21.15.26:0/4199939508 learned_addr learned my addr 172.21.15.26:0/4199939508 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e8a00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40fff70 0x7f89e40fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 -- 172.21.15.26:0/4199939508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40fff70 msgr2=0x7f89e40fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40fff70 0x7f89e40fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.607+0000 7f89e9400700 1 -- 172.21.15.26:0/4199939508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0005040 con 0x7f89e40662f0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89e9400700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89d8002a50 tx=0x7f89d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d8007cc0 con 0x7f89e40662f0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89d800f040 con 0x7f89e40662f0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e40fc330 con 0x7f89e40662f0 2024-04-16T07:19:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d800daf0 con 0x7f89e40662f0 2024-04-16T07:19:36.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.608+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e4115a50 con 0x7f89e40662f0 2024-04-16T07:19:36.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.610+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e405f5f0 con 0x7f89e40662f0 2024-04-16T07:19:36.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.611+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f89d8009810 con 0x7f89e40662f0 2024-04-16T07:19:36.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.611+0000 7f89e1600700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 0x7f89cc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:36.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.612+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f89d8086c00 con 0x7f89e40662f0 2024-04-16T07:19:36.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.612+0000 7f89e8a00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 0x7f89cc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:36.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.612+0000 7f89e8a00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 0x7f89cc06b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f89d000b970 tx=0x7f89d0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:36.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.616+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f89d808c020 con 0x7f89e40662f0 2024-04-16T07:19:36.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:36 smithi026 ceph-mon[28805]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:36 smithi104 ceph-mon[36163]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:36.867+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f89e40f8fb0 con 0x7f89cc0692b0 2024-04-16T07:19:37.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:37 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T07:19:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:37 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T07:19:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.570+0000 7f89e1600700 1 -- 172.21.15.26:0/4199939508 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f89e40f8fb0 con 0x7f89cc0692b0 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 msgr2=0x7f89cc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 0x7f89cc06b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f89d000b970 tx=0x7f89d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 msgr2=0x7f89e40fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89d8002a50 tx=0x7f89d8004600 comp rx=0 tx=0).stop 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 shutdown_connections 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f89cc0692b0 0x7f89cc06b760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89e40662f0 0x7f89e40fd460 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 --2- 172.21.15.26:0/4199939508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89e40fff70 0x7f89e40fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 >> 172.21.15.26:0/4199939508 conn(0x7f89e40f74c0 msgr2=0x7f89e40f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 shutdown_connections 2024-04-16T07:19:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:38.573+0000 7f89ebe00700 1 -- 172.21.15.26:0/4199939508 wait complete. 2024-04-16T07:19:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T07:19:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T07:19:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T07:19:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T07:19:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:38 smithi104 ceph-mon[36163]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[28800]: 2024-04-16T07:19:38.523+0000 7fec5f600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T07:19:38.942 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:38 smithi026 ceph-mon[28805]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T07:19:39.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: fsmap cephfs:0 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: Saving service mds.cephfs spec with placement count:4 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:39.800 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:39.801 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:39.801 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:39 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:39.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T07:19:39.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: fsmap cephfs:0 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: Saving service mds.cephfs spec with placement count:4 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:39.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:39 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:40.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:40 smithi026 ceph-mon[28805]: Deploying daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:19:40.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:40 smithi026 ceph-mon[28805]: pgmap v216: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:40.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:40 smithi026 ceph-mon[28805]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T07:19:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:40 smithi104 ceph-mon[36163]: Deploying daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:19:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:40 smithi104 ceph-mon[36163]: pgmap v216: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:40 smithi104 ceph-mon[36163]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T07:19:41.160 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:41.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:41 smithi104 ceph-mon[36163]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T07:19:41.865 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:41 smithi026 ceph-mon[28805]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T07:19:42.842 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:42 smithi026 ceph-mon[28805]: pgmap v219: 65 pgs: 8 creating+peering, 19 active+clean, 38 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:42 smithi104 ceph-mon[36163]: pgmap v219: 65 pgs: 8 creating+peering, 19 active+clean, 38 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 -- 172.21.15.26:0/2941101314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 msgr2=0x7fd3300fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 --2- 172.21.15.26:0/2941101314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd3300fd290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd320004940 tx=0x7fd320009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 -- 172.21.15.26:0/2941101314 shutdown_connections 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 --2- 172.21.15.26:0/2941101314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 0x7fd3300fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 --2- 172.21.15.26:0/2941101314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd3300fd290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 -- 172.21.15.26:0/2941101314 >> 172.21.15.26:0/2941101314 conn(0x7fd3300f8580 msgr2=0x7fd3300fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 -- 172.21.15.26:0/2941101314 shutdown_connections 2024-04-16T07:19:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.734+0000 7fd335a00700 1 -- 172.21.15.26:0/2941101314 wait complete. 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.735+0000 7fd335a00700 1 Processor -- start 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.735+0000 7fd335a00700 1 -- start start 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd335a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd335a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 0x7fd330070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd335a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330072260 con 0x7fd3300fce80 2024-04-16T07:19:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd335a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330070850 con 0x7fd3300fdff0 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56600/0 (socket says 172.21.15.26:56600) 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 -- 172.21.15.26:0/847678545 learned_addr learned my addr 172.21.15.26:0/847678545 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32e000700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 0x7fd330070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 -- 172.21.15.26:0/847678545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 msgr2=0x7fd330070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 0x7fd330070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.736+0000 7fd32ea00700 1 -- 172.21.15.26:0/847678545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320005040 con 0x7fd3300fce80 2024-04-16T07:19:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd32ea00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd320009150 tx=0x7fd320009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32000e040 con 0x7fd3300fce80 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd32000bcc0 con 0x7fd3300fce80 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32001c630 con 0x7fd3300fce80 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330070aa0 con 0x7fd3300fce80 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.737+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd330071060 con 0x7fd3300fce80 2024-04-16T07:19:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.738+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33004c710 con 0x7fd3300fce80 2024-04-16T07:19:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.739+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd320013420 con 0x7fd3300fce80 2024-04-16T07:19:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.741+0000 7fd326a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 0x7fd31406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.741+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd320087f40 con 0x7fd3300fce80 2024-04-16T07:19:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.741+0000 7fd32e000700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 0x7fd31406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.742+0000 7fd32e000700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 0x7fd31406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd31c004990 tx=0x7fd31c009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:43.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:43.746+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd320057ce0 con 0x7fd3300fce80 2024-04-16T07:19:44.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.014+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd330115c50 con 0x7fd3300fce80 2024-04-16T07:19:44.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.016+0000 7fd326a00700 1 -- 172.21.15.26:0/847678545 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1243 (secure 0 0 0) 0x7fd320057870 con 0x7fd3300fce80 2024-04-16T07:19:44.015 INFO:teuthology.orchestra.run.smithi026.stdout:e4 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 4 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:43.600421+0000 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:19:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 0 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14454} 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 0 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:14454} state up:creating seq 1 addr [v2:172.21.15.26:6826/29205890,v1:172.21.15.26:6827/29205890] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:19:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:19:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 msgr2=0x7fd31406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 0x7fd31406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd31c004990 tx=0x7fd31c009ee0 comp rx=0 tx=0).stop 2024-04-16T07:19:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 msgr2=0x7fd330071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd320009150 tx=0x7fd320009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 shutdown_connections 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd3140692a0 0x7fd31406b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3300fce80 0x7fd330071c60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 --2- 172.21.15.26:0/847678545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3300fdff0 0x7fd330070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.018+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 >> 172.21.15.26:0/847678545 conn(0x7fd3300f8580 msgr2=0x7fd3300f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.019+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 shutdown_connections 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:44.019+0000 7fd335a00700 1 -- 172.21.15.26:0/847678545 wait complete. 2024-04-16T07:19:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 4 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: pgmap v220: 65 pgs: 8 creating+peering, 43 active+clean, 14 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6826/29205890,v1:172.21.15.26:6827/29205890] up:boot 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: daemon mds.cephfs.smithi026.kqpyuh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: Cluster is now healthy 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: fsmap cephfs:0 1 up:standby 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:creating} 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: daemon mds.cephfs.smithi026.kqpyuh is now active in filesystem cephfs as rank 0 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:44.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:44.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:44.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:44 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/847678545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: pgmap v220: 65 pgs: 8 creating+peering, 43 active+clean, 14 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6826/29205890,v1:172.21.15.26:6827/29205890] up:boot 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: daemon mds.cephfs.smithi026.kqpyuh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: Cluster is now healthy 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: fsmap cephfs:0 1 up:standby 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:19:44.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:creating} 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: daemon mds.cephfs.smithi026.kqpyuh is now active in filesystem cephfs as rank 0 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:44.852 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:44 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/847678545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:19:44.900 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:44.913 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:44.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-16T07:19:45.427 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:45.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:45 smithi104 ceph-mon[36163]: Deploying daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:19:45.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:45 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6826/29205890,v1:172.21.15.26:6827/29205890] up:active 2024-04-16T07:19:45.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:45 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2024-04-16T07:19:45.907 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:45 smithi026 ceph-mon[28805]: Deploying daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:19:45.907 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:45 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6826/29205890,v1:172.21.15.26:6827/29205890] up:active 2024-04-16T07:19:45.907 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:45 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2024-04-16T07:19:46.620 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:46 smithi026 ceph-mon[28805]: pgmap v221: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 151 B/s wr, 0 op/s 2024-04-16T07:19:46.620 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:46.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:46 smithi104 ceph-mon[36163]: pgmap v221: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 151 B/s wr, 0 op/s 2024-04-16T07:19:46.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.600+0000 7efdbc400700 1 -- 172.21.15.26:0/2331949856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fe040 msgr2=0x7efdb40fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.600+0000 7efdbc400700 1 --2- 172.21.15.26:0/2331949856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fe040 0x7efdb40fe4b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efda0004940 tx=0x7efda0009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 -- 172.21.15.26:0/2331949856 shutdown_connections 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 --2- 172.21.15.26:0/2331949856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fe040 0x7efdb40fe4b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 --2- 172.21.15.26:0/2331949856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fce40 0x7efdb40fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 -- 172.21.15.26:0/2331949856 >> 172.21.15.26:0/2331949856 conn(0x7efdb40f8590 msgr2=0x7efdb40fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 -- 172.21.15.26:0/2331949856 shutdown_connections 2024-04-16T07:19:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.601+0000 7efdbc400700 1 -- 172.21.15.26:0/2331949856 wait complete. 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 Processor -- start 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 -- start start 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fe040 0x7efdb4107600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4107c50 con 0x7efdb40fce40 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.602+0000 7efdbc400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb41084a0 con 0x7efdb40fe040 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56618/0 (socket says 172.21.15.26:56618) 2024-04-16T07:19:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 -- 172.21.15.26:0/3665898107 learned_addr learned my addr 172.21.15.26:0/3665898107 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9000700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fe040 0x7efdb4107600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 -- 172.21.15.26:0/3665898107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fe040 msgr2=0x7efdb4107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fe040 0x7efdb4107600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 -- 172.21.15.26:0/3665898107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0005040 con 0x7efdb40fce40 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.603+0000 7efdb9a00700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efda8002a80 tx=0x7efda8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.604+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda8007cc0 con 0x7efdb40fce40 2024-04-16T07:19:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.604+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb4108780 con 0x7efdb40fce40 2024-04-16T07:19:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.604+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efda800f040 con 0x7efdb40fce40 2024-04-16T07:19:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.604+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda800dad0 con 0x7efdb40fce40 2024-04-16T07:19:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.604+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb4108d40 con 0x7efdb40fce40 2024-04-16T07:19:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.605+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb41019d0 con 0x7efdb40fce40 2024-04-16T07:19:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.606+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7efda800dc30 con 0x7efdb40fce40 2024-04-16T07:19:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.607+0000 7efdb1600700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 0x7efd9c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.608+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7efda8086ff0 con 0x7efdb40fce40 2024-04-16T07:19:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.608+0000 7efdb9000700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 0x7efd9c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.608+0000 7efdb9000700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 0x7efd9c06b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efda0004f90 tx=0x7efda0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.616+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7efda8053200 con 0x7efdb40fce40 2024-04-16T07:19:47.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:47.910+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7efdb405f5f0 con 0x7efdb40fce40 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:48.470 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: Deploying daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: pgmap v222: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 2 op/s 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:boot 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:48.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:48 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3665898107' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T07:19:48.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: Deploying daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: pgmap v222: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 2 op/s 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:boot 2024-04-16T07:19:48.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:48.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:19:48.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:48.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:48 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3665898107' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T07:19:48.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.626+0000 7efdb1600700 1 -- 172.21.15.26:0/3665898107 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7efda8056820 con 0x7efdb40fce40 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 msgr2=0x7efd9c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 0x7efd9c06b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efda0004f90 tx=0x7efda0009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 msgr2=0x7efdb41070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efda8002a80 tx=0x7efda8004600 comp rx=0 tx=0).stop 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 shutdown_connections 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd9c0692b0 0x7efd9c06b760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdb40fce40 0x7efdb41070c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 --2- 172.21.15.26:0/3665898107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdb40fe040 0x7efdb4107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.630+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 >> 172.21.15.26:0/3665898107 conn(0x7efdb40f8590 msgr2=0x7efdb40f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:48.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.631+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 shutdown_connections 2024-04-16T07:19:48.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:48.631+0000 7efdbc400700 1 -- 172.21.15.26:0/3665898107 wait complete. 2024-04-16T07:19:49.482 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:49.493 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:49.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-16T07:19:49.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:49 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3665898107' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T07:19:49.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:49 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:49.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:49 smithi026 ceph-mon[28805]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-16T07:19:50.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:49 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3665898107' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T07:19:50.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:49 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 1 up:standby 2024-04-16T07:19:50.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:49 smithi104 ceph-mon[36163]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-16T07:19:50.156 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:52 smithi026 ceph-mon[28805]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-16T07:19:52.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:52 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:52 smithi104 ceph-mon[36163]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-16T07:19:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:52 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 -- 172.21.15.26:0/1171874365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c4103420 msgr2=0x7f27c4103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 --2- 172.21.15.26:0/1171874365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c4103420 0x7f27c4103830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27b4004940 tx=0x7f27b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 -- 172.21.15.26:0/1171874365 shutdown_connections 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 --2- 172.21.15.26:0/1171874365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41046d0 0x7f27c4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 --2- 172.21.15.26:0/1171874365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c4103420 0x7f27c4103830 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.922+0000 7f27caa00700 1 -- 172.21.15.26:0/1171874365 >> 172.21.15.26:0/1171874365 conn(0x7f27c4072640 msgr2=0x7f27c4074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.923+0000 7f27caa00700 1 -- 172.21.15.26:0/1171874365 shutdown_connections 2024-04-16T07:19:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.923+0000 7f27caa00700 1 -- 172.21.15.26:0/1171874365 wait complete. 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 Processor -- start 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 -- start start 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4103420 0x7f27c4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4114aa0 con 0x7f27c41046d0 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.924+0000 7f27caa00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4114c10 con 0x7f27c4103420 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37452/0 (socket says 172.21.15.26:37452) 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 -- 172.21.15.26:0/1822542998 learned_addr learned my addr 172.21.15.26:0/1822542998 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3e00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4103420 0x7f27c4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 -- 172.21.15.26:0/1822542998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4103420 msgr2=0x7f27c4113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4103420 0x7f27c4113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.925+0000 7f27c3400700 1 -- 172.21.15.26:0/1822542998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4005040 con 0x7f27c41046d0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27c3400700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f27b0002a50 tx=0x7f27b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b0007cc0 con 0x7f27c41046d0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27b000f040 con 0x7f27c41046d0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c4119670 con 0x7f27c41046d0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b0009640 con 0x7f27c41046d0 2024-04-16T07:19:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.926+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c4119c30 con 0x7f27c41046d0 2024-04-16T07:19:52.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.927+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c404c710 con 0x7f27c41046d0 2024-04-16T07:19:52.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.928+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f27b00097a0 con 0x7f27c41046d0 2024-04-16T07:19:52.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.929+0000 7f27bbe00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 0x7f27a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:52.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.930+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f27b0086f60 con 0x7f27c41046d0 2024-04-16T07:19:52.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.930+0000 7f27c3e00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 0x7f27a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:52.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.931+0000 7f27c3e00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 0x7f27a806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27b400b3b0 tx=0x7f27b4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:52.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:52.934+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f27b0056c80 con 0x7f27c41046d0 2024-04-16T07:19:53.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.197+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f27c4119ea0 con 0x7f27c41046d0 2024-04-16T07:19:53.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.511+0000 7f27bbe00700 1 -- 172.21.15.26:0/1822542998 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f27c4119ea0 con 0x7f27c41046d0 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:boot 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: Deploying daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:19:53.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:53 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1822542998' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.515+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 msgr2=0x7f27a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.515+0000 7f27caa00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 0x7f27a806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27b400b3b0 tx=0x7f27b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.515+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 msgr2=0x7f27c41144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.515+0000 7f27caa00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f27b0002a50 tx=0x7f27b00046b0 comp rx=0 tx=0).stop 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 shutdown_connections 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f27a80692a0 0x7f27a806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4103420 0x7f27c4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 --2- 172.21.15.26:0/1822542998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c41046d0 0x7f27c41144a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 >> 172.21.15.26:0/1822542998 conn(0x7f27c4072640 msgr2=0x7f27c4073d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 shutdown_connections 2024-04-16T07:19:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:53.516+0000 7f27caa00700 1 -- 172.21.15.26:0/1822542998 wait complete. 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:boot 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T07:19:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:19:53.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: Deploying daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:19:53.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:53 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1822542998' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T07:19:54.181 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:54.191 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:54.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-16T07:19:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:54 smithi026 ceph-mon[28805]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T07:19:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:54 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1822542998' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T07:19:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:54 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:19:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:54 smithi104 ceph-mon[36163]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T07:19:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:54 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1822542998' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T07:19:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:54 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:56.609 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:56 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 -- 172.21.15.26:0/1197235690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fcef0 msgr2=0x7fc1a80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 --2- 172.21.15.26:0/1197235690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fcef0 0x7fc1a80fd320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc198004940 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 -- 172.21.15.26:0/1197235690 shutdown_connections 2024-04-16T07:19:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 --2- 172.21.15.26:0/1197235690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 --2- 172.21.15.26:0/1197235690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fcef0 0x7fc1a80fd320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 -- 172.21.15.26:0/1197235690 >> 172.21.15.26:0/1197235690 conn(0x7fc1a80f8530 msgr2=0x7fc1a80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 -- 172.21.15.26:0/1197235690 shutdown_connections 2024-04-16T07:19:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.767+0000 7fc1aea00700 1 -- 172.21.15.26:0/1197235690 wait complete. 2024-04-16T07:19:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.768+0000 7fc1aea00700 1 Processor -- start 2024-04-16T07:19:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.768+0000 7fc1aea00700 1 -- start start 2024-04-16T07:19:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc1aea00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fcef0 0x7fc1a810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc1aea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc1aea00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810e5c0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc1aea00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810ee10 con 0x7fc1a80fcef0 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37480/0 (socket says 172.21.15.26:37480) 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 -- 172.21.15.26:0/3189861531 learned_addr learned my addr 172.21.15.26:0/3189861531 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc1a7e00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fcef0 0x7fc1a810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 -- 172.21.15.26:0/3189861531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fcef0 msgr2=0x7fc1a810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fcef0 0x7fc1a810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.769+0000 7fc19fe00700 1 -- 172.21.15.26:0/3189861531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc19fe00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc190002a50 tx=0x7fc1900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc190007cc0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc19000f040 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a810f0f0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc190009640 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.770+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a810f630 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.771+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a805f5f0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.772+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc1900097a0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.773+0000 7fc1a4c00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 0x7fc18c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:19:56.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.773+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc190086fb0 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.774+0000 7fc1a7e00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 0x7fc18c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:19:56.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.774+0000 7fc1a7e00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 0x7fc18c06b760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc19800b3b0 tx=0x7fc198009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:19:56.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:56.778+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc190056c80 con 0x7fc1a80fe0e0 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:19:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:56 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:57.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.104+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7fc1a81059a0 con 0x7fc1a80fe0e0 2024-04-16T07:19:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.537+0000 7fc1a4c00700 1 -- 172.21.15.26:0/3189861531 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v13) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc190056810 con 0x7fc1a80fe0e0 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 msgr2=0x7fc18c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 0x7fc18c06b760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc19800b3b0 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 msgr2=0x7fc1a810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc190002a50 tx=0x7fc1900046b0 comp rx=0 tx=0).stop 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 shutdown_connections 2024-04-16T07:19:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fc18c0692b0 0x7fc18c06b760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a80fcef0 0x7fc1a810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 --2- 172.21.15.26:0/3189861531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1a80fe0e0 0x7fc1a810dfc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:19:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 >> 172.21.15.26:0/3189861531 conn(0x7fc1a80f8530 msgr2=0x7fc1a80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:19:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.541+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 shutdown_connections 2024-04-16T07:19:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:19:57.542+0000 7fc1aea00700 1 -- 172.21.15.26:0/3189861531 wait complete. 2024-04-16T07:19:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:inline data disabled 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:boot 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:standby 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: Dropping low affinity active daemon mds.cephfs.smithi026.kqpyuh in favor of higher affinity standby. 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: Replacing daemon mds.cephfs.smithi026.kqpyuh as rank 0 with standby daemon mds.cephfs.smithi026.kwjewe 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 3 up:standby 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:replay} 2 up:standby 2024-04-16T07:19:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:57 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3189861531' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T07:19:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:boot 2024-04-16T07:19:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:standby 2024-04-16T07:19:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: Dropping low affinity active daemon mds.cephfs.smithi026.kqpyuh in favor of higher affinity standby. 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: Replacing daemon mds.cephfs.smithi026.kqpyuh as rank 0 with standby daemon mds.cephfs.smithi026.kwjewe 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 3 up:standby 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:replay} 2 up:standby 2024-04-16T07:19:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:57 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3189861531' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T07:19:58.561 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:19:58.572 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:19:58.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 716 B/s wr, 2 op/s 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:reconnect 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] up:boot 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3189861531' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:reconnect} 3 up:standby 2024-04-16T07:19:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:58 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 716 B/s wr, 2 op/s 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:reconnect 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] up:boot 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3189861531' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:reconnect} 3 up:standby 2024-04-16T07:19:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:58 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:19:59.622 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:19:59.818 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:59 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:rejoin 2024-04-16T07:19:59.818 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:59 smithi026 ceph-mon[28805]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:rejoin} 3 up:standby 2024-04-16T07:19:59.818 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:59 smithi026 ceph-mon[28805]: daemon mds.cephfs.smithi026.kwjewe is now active in filesystem cephfs as rank 0 2024-04-16T07:19:59.818 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:59.818 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:19:59 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:59 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:rejoin 2024-04-16T07:19:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:59 smithi104 ceph-mon[36163]: fsmap cephfs:1/1 {0=cephfs.smithi026.kwjewe=up:rejoin} 3 up:standby 2024-04-16T07:19:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:59 smithi104 ceph-mon[36163]: daemon mds.cephfs.smithi026.kwjewe is now active in filesystem cephfs as rank 0 2024-04-16T07:19:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:19:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:19:59 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: Cluster is now healthy 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:active 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:standby 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:20:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:00 smithi104 ceph-mon[36163]: overall HEALTH_OK 2024-04-16T07:20:00.835 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:20:00.835 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:20:00.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: Cluster is now healthy 2024-04-16T07:20:00.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] up:active 2024-04-16T07:20:00.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:standby 2024-04-16T07:20:00.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:20:00.836 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:00 smithi026 ceph-mon[28805]: overall HEALTH_OK 2024-04-16T07:20:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:01 smithi104 ceph-mon[36163]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:standby 2024-04-16T07:20:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:01 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:20:01.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:01.916 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:01 smithi026 ceph-mon[28805]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:standby 2024-04-16T07:20:01.916 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:01 smithi026 ceph-mon[28805]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:20:01.916 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 -- 172.21.15.26:0/594476246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3080ff020 msgr2=0x7fa3080ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 --2- 172.21.15.26:0/594476246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3080ff020 0x7fa3080ff430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8004940 tx=0x7fa2f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 -- 172.21.15.26:0/594476246 shutdown_connections 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 --2- 172.21.15.26:0/594476246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3081002d0 0x7fa308100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 --2- 172.21.15.26:0/594476246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3080ff020 0x7fa3080ff430 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 -- 172.21.15.26:0/594476246 >> 172.21.15.26:0/594476246 conn(0x7fa3080fa950 msgr2=0x7fa3080fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 -- 172.21.15.26:0/594476246 shutdown_connections 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.611+0000 7fa30d400700 1 -- 172.21.15.26:0/594476246 wait complete. 2024-04-16T07:20:02.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 Processor -- start 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 -- start start 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 0x7fa30810fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308110750 con 0x7fa3081002d0 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.612+0000 7fa30d400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308110fa0 con 0x7fa3080ff020 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34448/0 (socket says 172.21.15.26:34448) 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 -- 172.21.15.26:0/1804037906 learned_addr learned my addr 172.21.15.26:0/1804037906 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa306800700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 0x7fa30810fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 -- 172.21.15.26:0/1804037906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 msgr2=0x7fa30810fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 0x7fa30810fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 -- 172.21.15.26:0/1804037906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8005040 con 0x7fa3081002d0 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa306800700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 0x7fa30810fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:20:02.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.613+0000 7fa2fde00700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c830 tx=0x7fa2f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.614+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f000a760 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.614+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2f000ad90 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.614+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f00113a0 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.614+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa308111280 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.614+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3081117c0 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.615+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30805f5f0 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.616+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa2f0012400 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.617+0000 7fa2ff400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 0x7fa2ec06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.617+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa2f0087290 con 0x7fa3081002d0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.618+0000 7fa306800700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 0x7fa2ec06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.618+0000 7fa306800700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 0x7fa2ec06b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800b7a0 tx=0x7fa2f8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:02.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.622+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa2f0057000 con 0x7fa3081002d0 2024-04-16T07:20:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:02 smithi104 ceph-mon[36163]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-04-16T07:20:02.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.912+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa308111b90 con 0x7fa3081002d0 2024-04-16T07:20:02.911 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:02 smithi026 ceph-mon[28805]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-04-16T07:20:02.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.913+0000 7fa2ff400700 1 -- 172.21.15.26:0/1804037906 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7fa2f0056b90 con 0x7fa3081002d0 2024-04-16T07:20:02.913 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:20:02.913 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:20:02.913 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:20:02.913 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:20:02.914 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:20:02.915 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:02.916 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 msgr2=0x7fa2ec06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 0x7fa2ec06b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800b7a0 tx=0x7fa2f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 msgr2=0x7fa308110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c830 tx=0x7fa2f0004470 comp rx=0 tx=0).stop 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 shutdown_connections 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2ec0692b0 0x7fa2ec06b760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3080ff020 0x7fa30810fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 --2- 172.21.15.26:0/1804037906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3081002d0 0x7fa308110150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 >> 172.21.15.26:0/1804037906 conn(0x7fa3080fa950 msgr2=0x7fa3080fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 shutdown_connections 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:02.916+0000 7fa30d400700 1 -- 172.21.15.26:0/1804037906 wait complete. 2024-04-16T07:20:02.917 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:20:03.854 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:03 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1804037906' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:20:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:03 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1804037906' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:20:04.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-16T07:20:04.848 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:04.992 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:04 smithi026 ceph-mon[28805]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-04-16T07:20:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:04 smithi104 ceph-mon[36163]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.005 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:20:06.006 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:05 smithi026 ceph-mon[28805]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-04-16T07:20:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:20:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:05 smithi104 ceph-mon[36163]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-04-16T07:20:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 -- 172.21.15.26:0/3011505371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 msgr2=0x7ff1d00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 --2- 172.21.15.26:0/3011505371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d00fd320 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0004940 tx=0x7ff1c0009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 -- 172.21.15.26:0/3011505371 shutdown_connections 2024-04-16T07:20:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 --2- 172.21.15.26:0/3011505371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 0x7ff1d00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 --2- 172.21.15.26:0/3011505371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d00fd320 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 -- 172.21.15.26:0/3011505371 >> 172.21.15.26:0/3011505371 conn(0x7ff1d00f8530 msgr2=0x7ff1d00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 -- 172.21.15.26:0/3011505371 shutdown_connections 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.885+0000 7ff1d6000700 1 -- 172.21.15.26:0/3011505371 wait complete. 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.886+0000 7ff1d6000700 1 Processor -- start 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.886+0000 7ff1d6000700 1 -- start start 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1d6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1d6000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 0x7ff1d0110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1d6000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d0110820 con 0x7ff1d00fcef0 2024-04-16T07:20:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1d6000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d0111010 con 0x7ff1d00fe0e0 2024-04-16T07:20:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34472/0 (socket says 172.21.15.26:34472) 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 -- 172.21.15.26:0/1841018027 learned_addr learned my addr 172.21.15.26:0/1841018027 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cea00700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 0x7ff1d0110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 -- 172.21.15.26:0/1841018027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 msgr2=0x7ff1d0110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 0x7ff1d0110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.887+0000 7ff1cf400700 1 -- 172.21.15.26:0/1841018027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c0005040 con 0x7ff1d00fcef0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1cf400700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000b970 tx=0x7ff1c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c000e040 con 0x7ff1d00fcef0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d0111290 con 0x7ff1d00fcef0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1c0004290 con 0x7ff1d00fcef0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c001c630 con 0x7ff1d00fcef0 2024-04-16T07:20:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.888+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d0111850 con 0x7ff1d00fcef0 2024-04-16T07:20:06.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.889+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d0101a70 con 0x7ff1d00fcef0 2024-04-16T07:20:06.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.891+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff1c000bbf0 con 0x7ff1d00fcef0 2024-04-16T07:20:06.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.892+0000 7ff1c7400700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 0x7ff1b406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:06.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.892+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff1c00882b0 con 0x7ff1d00fcef0 2024-04-16T07:20:06.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.893+0000 7ff1cea00700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 0x7ff1b406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:06.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.893+0000 7ff1cea00700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 0x7ff1b406b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc006fd0 tx=0x7ff1bc008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:06.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:06.898+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff1c00580c0 con 0x7ff1d00fcef0 2024-04-16T07:20:07.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.174+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1d005f5f0 con 0x7ff1d00fcef0 2024-04-16T07:20:07.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.175+0000 7ff1c7400700 1 -- 172.21.15.26:0/1841018027 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4691 (secure 0 0 0) 0x7ff1c0013e60 con 0x7ff1d00fcef0 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 msgr2=0x7ff1b406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 0x7ff1b406b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc006fd0 tx=0x7ff1bc008040 comp rx=0 tx=0).stop 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 msgr2=0x7ff1d010fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000b970 tx=0x7ff1c0009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 shutdown_connections 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff1b40692b0 0x7ff1b406b760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1d00fcef0 0x7ff1d010fc50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 --2- 172.21.15.26:0/1841018027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d00fe0e0 0x7ff1d0110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 >> 172.21.15.26:0/1841018027 conn(0x7ff1d00f8530 msgr2=0x7ff1d00f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 shutdown_connections 2024-04-16T07:20:07.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:07.178+0000 7ff1d6000700 1 -- 172.21.15.26:0/1841018027 wait complete. 2024-04-16T07:20:07.178 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:20:07.193 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:20:07.276 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:07 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1841018027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:20:07.459 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:07 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1841018027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:20:08.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-16T07:20:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:08 smithi026 ceph-mon[28805]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T07:20:08.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:08 smithi104 ceph-mon[36163]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T07:20:09.280 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:09.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:09.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:09.604 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:09.604 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:10 smithi026 ceph-mon[28805]: pgmap v234: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T07:20:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:10 smithi104 ceph-mon[36163]: pgmap v234: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 -- 172.21.15.26:0/1710625552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 msgr2=0x7f26900ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 --2- 172.21.15.26:0/1710625552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f26900ffa70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2680009ce0 tx=0x7f2680004d10 comp rx=0 tx=0).stop 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 -- 172.21.15.26:0/1710625552 shutdown_connections 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 --2- 172.21.15.26:0/1710625552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f2690102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 --2- 172.21.15.26:0/1710625552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f26900ffa70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 -- 172.21.15.26:0/1710625552 >> 172.21.15.26:0/1710625552 conn(0x7f26900f74c0 msgr2=0x7f26900f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 -- 172.21.15.26:0/1710625552 shutdown_connections 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.971+0000 7f2695600700 1 -- 172.21.15.26:0/1710625552 wait complete. 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.972+0000 7f2695600700 1 Processor -- start 2024-04-16T07:20:11.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f2695600700 1 -- start start 2024-04-16T07:20:11.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f2695600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:11.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f2695600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f26901103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:11.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f2695600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690111110 con 0x7f26900662b0 2024-04-16T07:20:11.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f2695600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690111280 con 0x7f26900fffb0 2024-04-16T07:20:11.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f268ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:11.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f268ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56922/0 (socket says 172.21.15.26:56922) 2024-04-16T07:20:11.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.973+0000 7f268ea00700 1 -- 172.21.15.26:0/1698234133 learned_addr learned my addr 172.21.15.26:0/1698234133 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:11.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f2686000700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f26901103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:11.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f268ea00700 1 -- 172.21.15.26:0/1698234133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 msgr2=0x7f26901103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f268ea00700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f26901103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f268ea00700 1 -- 172.21.15.26:0/1698234133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009940 con 0x7f26900662b0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f2686000700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f26901103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.974+0000 7f268ea00700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2680009380 tx=0x7f2680004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.975+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268000e040 con 0x7f26900662b0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.975+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26800128e0 con 0x7f26900662b0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.975+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26901114a0 con 0x7f26900662b0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.975+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268001c9f0 con 0x7f26900662b0 2024-04-16T07:20:11.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.975+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2690111a60 con 0x7f26900662b0 2024-04-16T07:20:11.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.977+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26900f8ff0 con 0x7f26900662b0 2024-04-16T07:20:11.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.977+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2680012a50 con 0x7f26900662b0 2024-04-16T07:20:11.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.979+0000 7f2687400700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 0x7f267406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:11.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.979+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2680088eb0 con 0x7f26900662b0 2024-04-16T07:20:11.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.979+0000 7f2686000700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 0x7f267406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:11.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.980+0000 7f2686000700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 0x7f267406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2678007fd0 tx=0x7f2678009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:11.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:11.984+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2680055110 con 0x7f26900662b0 2024-04-16T07:20:12.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.305+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f269005f5f0 con 0x7f26900662b0 2024-04-16T07:20:12.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.306+0000 7f2687400700 1 -- 172.21.15.26:0/1698234133 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f2680058730 con 0x7f26900662b0 2024-04-16T07:20:12.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.310+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 msgr2=0x7f267406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.310+0000 7f2695600700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 0x7f267406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2678007fd0 tx=0x7f2678009040 comp rx=0 tx=0).stop 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.310+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 msgr2=0x7f269010fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.310+0000 7f2695600700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2680009380 tx=0x7f2680004d10 comp rx=0 tx=0).stop 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 shutdown_connections 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f26740692a0 0x7f267406b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26900662b0 0x7f269010fe60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 --2- 172.21.15.26:0/1698234133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26900fffb0 0x7f26901103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 >> 172.21.15.26:0/1698234133 conn(0x7f26900f74c0 msgr2=0x7f26900fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 shutdown_connections 2024-04-16T07:20:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:12.311+0000 7f2695600700 1 -- 172.21.15.26:0/1698234133 wait complete. 2024-04-16T07:20:12.328 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:20:12.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:12 smithi026 ceph-mon[28805]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T07:20:12.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:12 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1698234133' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T07:20:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:12 smithi104 ceph-mon[36163]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T07:20:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:12 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1698234133' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T07:20:13.395 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-16T07:20:13.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:20:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 -- 172.21.15.26:0/3888374110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 msgr2=0x7fde78103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 --2- 172.21.15.26:0/3888374110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde78103ab0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fde640048d0 tx=0x7fde64009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 -- 172.21.15.26:0/3888374110 shutdown_connections 2024-04-16T07:20:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 --2- 172.21.15.26:0/3888374110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde78103ab0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 --2- 172.21.15.26:0/3888374110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 0x7fde78101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 -- 172.21.15.26:0/3888374110 >> 172.21.15.26:0/3888374110 conn(0x7fde780f6860 msgr2=0x7fde780f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 -- 172.21.15.26:0/3888374110 shutdown_connections 2024-04-16T07:20:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.672+0000 7fde7d400640 1 -- 172.21.15.26:0/3888374110 wait complete. 2024-04-16T07:20:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.673+0000 7fde7d400640 1 Processor -- start 2024-04-16T07:20:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.673+0000 7fde7d400640 1 -- start start 2024-04-16T07:20:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde7d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 0x7fde7810eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde7d400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde7d400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7810fa40 con 0x7fde781016d0 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde7d400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78114440 con 0x7fde78066450 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56936/0 (socket says 172.21.15.26:56936) 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 -- 172.21.15.26:0/4009987069 learned_addr learned my addr 172.21.15.26:0/4009987069 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 -- 172.21.15.26:0/4009987069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 msgr2=0x7fde7810eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde76800640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 0x7fde7810eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 0x7fde7810eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.674+0000 7fde75e00640 1 -- 172.21.15.26:0/4009987069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64005040 con 0x7fde781016d0 2024-04-16T07:20:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde75e00640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde64000c00 tx=0x7fde64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde640094c0 con 0x7fde781016d0 2024-04-16T07:20:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde78114580 con 0x7fde781016d0 2024-04-16T07:20:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde64009620 con 0x7fde781016d0 2024-04-16T07:20:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde640435f0 con 0x7fde781016d0 2024-04-16T07:20:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.675+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde78114a10 con 0x7fde781016d0 2024-04-16T07:20:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.676+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde78064380 con 0x7fde781016d0 2024-04-16T07:20:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.678+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fde6402f070 con 0x7fde781016d0 2024-04-16T07:20:13.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.678+0000 7fde6ea00640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:13.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.679+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fde640a9290 con 0x7fde781016d0 2024-04-16T07:20:13.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.679+0000 7fde76800640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:13.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.680+0000 7fde76800640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 0x7fde5c06b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde68007fd0 tx=0x7fde68009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.683+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fde640754f0 con 0x7fde781016d0 2024-04-16T07:20:13.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.879+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde78106ed0 con 0x7fde781016d0 2024-04-16T07:20:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.882+0000 7fde6ea00640 1 -- 172.21.15.26:0/4009987069 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14396 (secure 0 0 0) 0x7fde64078b10 con 0x7fde781016d0 2024-04-16T07:20:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:13.881 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":47,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:19:56.515839+0000","last_up_change":"2024-04-16T07:18:03.334452+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T07:19:37.442413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T07:19:38.080594+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6803","nonce":3225147345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6805","nonce":3225147345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6809","nonce":3225147345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":3225147345},{"type":"v1","addr":"172.21.15.26:6807","nonce":3225147345}]},"public_addr":"172.21.15.26:6803/3225147345","cluster_addr":"172.21.15.26:6805/3225147345","heartbeat_back_addr":"172.21.15.26:6809/3225147345","heartbeat_front_addr":"172.21.15.26:6807/3225147345","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6811","nonce":1194632492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6813","nonce":1194632492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6817","nonce":1194632492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":1194632492},{"type":"v1","addr":"172.21.15.26:6815","nonce":1194632492}]},"public_addr":"172.21.15.26:6811/1194632492","cluster_addr":"172.21.15.26:6813/1194632492","heartbeat_back_addr":"172.21.15.26:6817/1194632492","heartbeat_front_addr":"172.21.15.26:6815/1194632492","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6819","nonce":828513924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6821","nonce":828513924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6825","nonce":828513924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":828513924},{"type":"v1","addr":"172.21.15.26:6823","nonce":828513924}]},"public_addr":"172.21.15.26:6819/828513924","cluster_addr":"172.21.15.26:6821/828513924","heartbeat_back_addr":"172.21.15.26:6825/828513924","heartbeat_front_addr":"172.21.15.26:6823/828513924","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6801","nonce":2432746791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6803","nonce":2432746791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6807","nonce":2432746791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2432746791},{"type":"v1","addr":"172.21.15.104:6805","nonce":2432746791}]},"public_addr":"172.21.15.104:6801/2432746791","cluster_addr":"172.21.15.104:6803/2432746791","heartbeat_back_addr":"172.21.15.104:6807/2432746791","heartbeat_front_addr":"172.21.15.104:6805/2432746791","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6809","nonce":3400087284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6811","nonce":3400087284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6815","nonce":3400087284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3400087284},{"type":"v1","addr":"172.21.15.104:6813","nonce":3400087284}]},"public_addr":"172.21.15.104:6809/3400087284","cluster_addr":"172.21.15.104:6811/3400087284","heartbeat_back_addr":"172.21.15.104:6815/3400087284","heartbeat_front_addr":"172.21.15.104:6813/3400087284","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6817","nonce":4159292069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6819","nonce":4159292069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6823","nonce":4159292069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4159292069},{"type":"v1","addr":"172.21.15.104:6821","nonce":4159292069}]},"public_addr":"172.21.15.104:6817/4159292069","cluster_addr":"172.21.15.104:6819/4159292069","heartbeat_back_addr":"172.21.15.104:6823/4159292069","heartbeat_front_addr":"172.21.15.104:6821/4159292069","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6827/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6826/29205890":"2024-04-17T07:19:56.515804+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:20:13.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.884+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 msgr2=0x7fde5c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:13.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.884+0000 7fde7d400640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 0x7fde5c06b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde68007fd0 tx=0x7fde68009040 comp rx=0 tx=0).stop 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.884+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 msgr2=0x7fde7810f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.884+0000 7fde7d400640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde64000c00 tx=0x7fde64009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 shutdown_connections 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde781016d0 0x7fde7810f3f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 --2- 172.21.15.26:0/4009987069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde78066450 0x7fde7810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 >> 172.21.15.26:0/4009987069 conn(0x7fde780f6860 msgr2=0x7fde780f6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 shutdown_connections 2024-04-16T07:20:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:13.885+0000 7fde7d400640 1 -- 172.21.15.26:0/4009987069 wait complete. 2024-04-16T07:20:13.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T07:16:23.932743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T07:19:37.442413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T07:19:38.080594+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-16T07:20:13.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 -- 172.21.15.26:0/1842139581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0fe240 msgr2=0x7f974c0fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 --2- 172.21.15.26:0/1842139581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0fe240 0x7f974c0fe650 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f973c0048d0 tx=0x7f973c009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 -- 172.21.15.26:0/1842139581 shutdown_connections 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 --2- 172.21.15.26:0/1842139581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0ff470 0x7f974c0ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 --2- 172.21.15.26:0/1842139581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0fe240 0x7f974c0fe650 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 -- 172.21.15.26:0/1842139581 >> 172.21.15.26:0/1842139581 conn(0x7f974c0f9ae0 msgr2=0x7f974c0fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:14.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 -- 172.21.15.26:0/1842139581 shutdown_connections 2024-04-16T07:20:14.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.141+0000 7f9751400640 1 -- 172.21.15.26:0/1842139581 wait complete. 2024-04-16T07:20:14.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 Processor -- start 2024-04-16T07:20:14.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 -- start start 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0fe240 0x7f974c10ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c10f8f0 con 0x7f974c0ff470 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.142+0000 7f9751400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1142f0 con 0x7f974c0fe240 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56960/0 (socket says 172.21.15.26:56960) 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 -- 172.21.15.26:0/2353849485 learned_addr learned my addr 172.21.15.26:0/2353849485 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974be00640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0fe240 0x7f974c10ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 -- 172.21.15.26:0/2353849485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0fe240 msgr2=0x7f974c10ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0fe240 0x7f974c10ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.143+0000 7f974b400640 1 -- 172.21.15.26:0/2353849485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c005040 con 0x7f974c0ff470 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f974b400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f973800c790 tx=0x7f97380043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973800a7a0 con 0x7f974c0ff470 2024-04-16T07:20:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f973800add0 con 0x7f974c0ff470 2024-04-16T07:20:14.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c114490 con 0x7f974c0ff470 2024-04-16T07:20:14.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97380115e0 con 0x7f974c0ff470 2024-04-16T07:20:14.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.144+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c114980 con 0x7f974c0ff470 2024-04-16T07:20:14.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.145+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c04bcc0 con 0x7f974c0ff470 2024-04-16T07:20:14.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.146+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f973800a900 con 0x7f974c0ff470 2024-04-16T07:20:14.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.147+0000 7f9743e00640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 0x7f973006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.147+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9738087380 con 0x7f974c0ff470 2024-04-16T07:20:14.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.148+0000 7f974be00640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 0x7f973006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.148+0000 7f974be00640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 0x7f973006b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f973c02bda0 tx=0x7f973c009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:14.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.152+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9738053660 con 0x7f974c0ff470 2024-04-16T07:20:14.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.356+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f974c114c30 con 0x7f974c0ff470 2024-04-16T07:20:14.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.357+0000 7f9743e00640 1 -- 172.21.15.26:0/2353849485 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9738056c80 con 0x7f974c0ff470 2024-04-16T07:20:14.356 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-16T07:20:14.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 msgr2=0x7f973006b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 0x7f973006b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f973c02bda0 tx=0x7f973c009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 msgr2=0x7f974c10f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f973800c790 tx=0x7f97380043c0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 shutdown_connections 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9730069220 0x7f973006b6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f974c0ff470 0x7f974c10f2a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 --2- 172.21.15.26:0/2353849485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f974c0fe240 0x7f974c10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 >> 172.21.15.26:0/2353849485 conn(0x7f974c0f9ae0 msgr2=0x7f974c0fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.360+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 shutdown_connections 2024-04-16T07:20:14.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.361+0000 7f9751400640 1 -- 172.21.15.26:0/2353849485 wait complete. 2024-04-16T07:20:14.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T07:20:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 -- 172.21.15.26:0/1306930080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 msgr2=0x7efd800fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 --2- 172.21.15.26:0/1306930080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd800fee50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efd700048d0 tx=0x7efd70009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 -- 172.21.15.26:0/1306930080 shutdown_connections 2024-04-16T07:20:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 --2- 172.21.15.26:0/1306930080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 0x7efd80101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 --2- 172.21.15.26:0/1306930080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd800fee50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 -- 172.21.15.26:0/1306930080 >> 172.21.15.26:0/1306930080 conn(0x7efd800f6880 msgr2=0x7efd800f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:14.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 -- 172.21.15.26:0/1306930080 shutdown_connections 2024-04-16T07:20:14.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.579+0000 7efd86e00640 1 -- 172.21.15.26:0/1306930080 wait complete. 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 Processor -- start 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 -- start start 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 0x7efd8010f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8010fb10 con 0x7efd80066180 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.580+0000 7efd86e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd80114510 con 0x7efd800ff390 2024-04-16T07:20:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56980/0 (socket says 172.21.15.26:56980) 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 -- 172.21.15.26:0/1178172085 learned_addr learned my addr 172.21.15.26:0/1178172085 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7f400640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 0x7efd8010f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 -- 172.21.15.26:0/1178172085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 msgr2=0x7efd8010f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 0x7efd8010f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.581+0000 7efd7fe00640 1 -- 172.21.15.26:0/1178172085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70005040 con 0x7efd80066180 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd7fe00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efd70000c00 tx=0x7efd70009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd700094c0 con 0x7efd80066180 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd70009620 con 0x7efd80066180 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd80114650 con 0x7efd80066180 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd700435f0 con 0x7efd80066180 2024-04-16T07:20:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.582+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd80114ae0 con 0x7efd80066180 2024-04-16T07:20:14.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.583+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8004bcc0 con 0x7efd80066180 2024-04-16T07:20:14.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.585+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7efd7002f070 con 0x7efd80066180 2024-04-16T07:20:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.586+0000 7efd77e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 0x7efd6406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.586+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7efd700a9320 con 0x7efd80066180 2024-04-16T07:20:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.586+0000 7efd7f400640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 0x7efd6406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:14.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.587+0000 7efd7f400640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 0x7efd6406b6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd80110480 tx=0x7efd6c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:14.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.591+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7efd700754e0 con 0x7efd80066180 2024-04-16T07:20:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:14 smithi026 ceph-mon[28805]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T07:20:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:14 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4009987069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:20:14.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.789+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7efd800f8320 con 0x7efd80066180 2024-04-16T07:20:14.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.789+0000 7efd77e00640 1 -- 172.21.15.26:0/1178172085 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7efd70078b00 con 0x7efd80066180 2024-04-16T07:20:14.788 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:20:14.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.791+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 msgr2=0x7efd6406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 0x7efd6406b6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd80110480 tx=0x7efd6c009040 comp rx=0 tx=0).stop 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 msgr2=0x7efd8010ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efd70000c00 tx=0x7efd70009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 shutdown_connections 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7efd64069220 0x7efd6406b6d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd800ff390 0x7efd8010f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 --2- 172.21.15.26:0/1178172085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd80066180 0x7efd8010ef80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 >> 172.21.15.26:0/1178172085 conn(0x7efd800f6880 msgr2=0x7efd800faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 shutdown_connections 2024-04-16T07:20:14.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:14.792+0000 7efd86e00640 1 -- 172.21.15.26:0/1178172085 wait complete. 2024-04-16T07:20:14.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T07:20:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:14 smithi104 ceph-mon[36163]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T07:20:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:14 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/4009987069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:20:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 -- 172.21.15.26:0/1442965249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 msgr2=0x7f265c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 --2- 172.21.15.26:0/1442965249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c070270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f26480048d0 tx=0x7f2648009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 -- 172.21.15.26:0/1442965249 shutdown_connections 2024-04-16T07:20:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 --2- 172.21.15.26:0/1442965249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c070270 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 --2- 172.21.15.26:0/1442965249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c06f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 -- 172.21.15.26:0/1442965249 >> 172.21.15.26:0/1442965249 conn(0x7f265c0f7820 msgr2=0x7f265c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 -- 172.21.15.26:0/1442965249 shutdown_connections 2024-04-16T07:20:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.003+0000 7f2663000640 1 -- 172.21.15.26:0/1442965249 wait complete. 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.004+0000 7f2663000640 1 Processor -- start 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.004+0000 7f2663000640 1 -- start start 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2663000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2663000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c10f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2663000640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c10f900 con 0x7f265c06fe00 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2663000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c114300 con 0x7f265c071390 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57004/0 (socket says 172.21.15.26:57004) 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 -- 172.21.15.26:0/4150552353 learned_addr learned my addr 172.21.15.26:0/4150552353 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661200640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c10f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 -- 172.21.15.26:0/4150552353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 msgr2=0x7f265c10f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c10f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661c00640 1 -- 172.21.15.26:0/4150552353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2648005040 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.005+0000 7f2661200640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c10f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2661c00640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2650002840 tx=0x7f2650004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2650007c30 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265000f040 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c1144a0 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2650009590 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.006+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c114990 con 0x7f265c06fe00 2024-04-16T07:20:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.007+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c04bcc0 con 0x7f265c06fe00 2024-04-16T07:20:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.008+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f265000d6b0 con 0x7f265c06fe00 2024-04-16T07:20:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.009+0000 7f2659600640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 0x7f264406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.009+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2650087120 con 0x7f265c06fe00 2024-04-16T07:20:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.010+0000 7f2661200640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 0x7f264406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.011+0000 7f2661200640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 0x7f264406b6d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f26480048a0 tx=0x7f2648009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.014+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2650053360 con 0x7f265c06fe00 2024-04-16T07:20:15.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.213+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f265c106dc0 con 0x7f265c06fe00 2024-04-16T07:20:15.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.214+0000 7f2659600640 1 -- 172.21.15.26:0/4150552353 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2650056980 con 0x7f265c06fe00 2024-04-16T07:20:15.213 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 msgr2=0x7f264406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 0x7f264406b6d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f26480048a0 tx=0x7f2648009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 msgr2=0x7f265c10ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2650002840 tx=0x7f2650004570 comp rx=0 tx=0).stop 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 shutdown_connections 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f2644069220 0x7f264406b6d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f265c071390 0x7f265c10f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 --2- 172.21.15.26:0/4150552353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f265c06fe00 0x7f265c10ed70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 >> 172.21.15.26:0/4150552353 conn(0x7f265c0f7820 msgr2=0x7f265c0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 shutdown_connections 2024-04-16T07:20:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.217+0000 7f2663000640 1 -- 172.21.15.26:0/4150552353 wait complete. 2024-04-16T07:20:15.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.445+0000 7f99aa400640 1 -- 172.21.15.26:0/3973203846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 msgr2=0x7f99a4073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.445+0000 7f99aa400640 1 --2- 172.21.15.26:0/3973203846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4073050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f99900048d0 tx=0x7f9990009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 -- 172.21.15.26:0/3973203846 shutdown_connections 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 --2- 172.21.15.26:0/3973203846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4073050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 --2- 172.21.15.26:0/3973203846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 0x7f99a4072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 -- 172.21.15.26:0/3973203846 >> 172.21.15.26:0/3973203846 conn(0x7f99a40f9c80 msgr2=0x7f99a40fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 -- 172.21.15.26:0/3973203846 shutdown_connections 2024-04-16T07:20:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.446+0000 7f99aa400640 1 -- 172.21.15.26:0/3973203846 wait complete. 2024-04-16T07:20:15.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 Processor -- start 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 -- start start 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 0x7f99a4113320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a4113eb0 con 0x7f99a4072660 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99aa400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41188b0 con 0x7f99a4071d10 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.447+0000 7f99a2a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57024/0 (socket says 172.21.15.26:57024) 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 -- 172.21.15.26:0/2439780483 learned_addr learned my addr 172.21.15.26:0/2439780483 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a3400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 0x7f99a4113320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 -- 172.21.15.26:0/2439780483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 msgr2=0x7f99a4113320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 0x7f99a4113320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 -- 172.21.15.26:0/2439780483 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990005040 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.448+0000 7f99a2a00640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9990000c00 tx=0x7f9990009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.449+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99900094c0 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.449+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9990032e90 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.449+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99900435f0 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.449+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a41189f0 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.449+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a4118e80 con 0x7f99a4072660 2024-04-16T07:20:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.451+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a4064380 con 0x7f99a4072660 2024-04-16T07:20:15.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.451+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f999002f070 con 0x7f99a4072660 2024-04-16T07:20:15.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.453+0000 7f999b400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 0x7f998806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:15.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.453+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f99900a9090 con 0x7f99a4072660 2024-04-16T07:20:15.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.453+0000 7f99a3400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 0x7f998806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:15.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.454+0000 7f99a3400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 0x7f998806b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f99a4071a70 tx=0x7f999400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:15.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.458+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9990075300 con 0x7f99a4072660 2024-04-16T07:20:15.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:15 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2353849485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:15.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:15.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:15 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1178172085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:15 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4150552353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.696+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99a410b340 con 0x7f99a4072660 2024-04-16T07:20:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.697+0000 7f999b400640 1 -- 172.21.15.26:0/2439780483 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4691 (secure 0 0 0) 0x7f9990078920 con 0x7f99a4072660 2024-04-16T07:20:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:15.696 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14480,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/4271222482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4271222482},{"type":"v1","addr":"172.21.15.26:6827","nonce":4271222482}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:19:59.548919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi026.kwjewe","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.26:6829/1284848687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1284848687},{"type":"v1","addr":"172.21.15.26:6829","nonce":1284848687}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-16T07:20:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 msgr2=0x7f998806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 0x7f998806b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f99a4071a70 tx=0x7f999400b040 comp rx=0 tx=0).stop 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 msgr2=0x7f99a4113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9990000c00 tx=0x7f9990009da0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 shutdown_connections 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9988069210 0x7f998806b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99a4072660 0x7f99a4113860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 --2- 172.21.15.26:0/2439780483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a4071d10 0x7f99a4113320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 >> 172.21.15.26:0/2439780483 conn(0x7f99a40f9c80 msgr2=0x7f99a40fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 shutdown_connections 2024-04-16T07:20:15.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:15.700+0000 7f99aa400640 1 -- 172.21.15.26:0/2439780483 wait complete. 2024-04-16T07:20:15.718 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-04-16T07:20:15.718 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-16T07:20:15.767 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-16T07:20:15.767 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-16T07:20:15.767 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-16T07:20:15.768 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T07:20:15.768 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:15.768 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-16T07:20:15.768 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T07:20:15.768 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:15.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:15.768 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-04-16T07:20:15.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:15.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link delete ceph-brx 2024-04-16T07:20:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:15 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2353849485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:15.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:15.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:15 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1178172085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:15 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/4150552353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:20:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:Cannot find device "ceph-brx" 2024-04-16T07:20:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:20:15.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:15.928 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-04-16T07:20:15.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:15.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-04-16T07:20:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:Cannot find device "ceph-brx" 2024-04-16T07:20:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:20:16.085 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-16T07:20:16.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2439780483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:20:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2439780483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:20:16.700 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:18.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:18 smithi026 ceph-mon[28805]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T07:20:18.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 -- 172.21.15.26:0/3839817529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd418102e60 msgr2=0x7fd418103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:18.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3839817529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd418102e60 0x7fd418103230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd40c004940 tx=0x7fd40c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 -- 172.21.15.26:0/3839817529 shutdown_connections 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3839817529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4180656b0 0x7fd418065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3839817529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd418102e60 0x7fd418103230 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 -- 172.21.15.26:0/3839817529 >> 172.21.15.26:0/3839817529 conn(0x7fd418072bc0 msgr2=0x7fd418072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 -- 172.21.15.26:0/3839817529 shutdown_connections 2024-04-16T07:20:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.724+0000 7fd41fe00700 1 -- 172.21.15.26:0/3839817529 wait complete. 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.725+0000 7fd41fe00700 1 Processor -- start 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.725+0000 7fd41fe00700 1 -- start start 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41fe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd418102e60 0x7fd4181162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41fe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4181168f0 con 0x7fd4180656b0 2024-04-16T07:20:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41fe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41810fcc0 con 0x7fd418102e60 2024-04-16T07:20:18.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:18.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50814/0 (socket says 172.21.15.26:50814) 2024-04-16T07:20:18.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.726+0000 7fd41d400700 1 -- 172.21.15.26:0/3991147333 learned_addr learned my addr 172.21.15.26:0/3991147333 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:18.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.727+0000 7fd41ca00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd418102e60 0x7fd4181162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.727+0000 7fd41d400700 1 -- 172.21.15.26:0/3991147333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd418102e60 msgr2=0x7fd4181162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.727+0000 7fd41d400700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd418102e60 0x7fd4181162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.727+0000 7fd41d400700 1 -- 172.21.15.26:0/3991147333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c005040 con 0x7fd4180656b0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.727+0000 7fd41d400700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009150 tx=0x7fd40c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.728+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd40c00e040 con 0x7fd4180656b0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.728+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd40c013420 con 0x7fd4180656b0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.728+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41810ff40 con 0x7fd4180656b0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.728+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd40c01c630 con 0x7fd4180656b0 2024-04-16T07:20:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.728+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd418110500 con 0x7fd4180656b0 2024-04-16T07:20:18.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.729+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd418107320 con 0x7fd4180656b0 2024-04-16T07:20:18.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.730+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd40c013590 con 0x7fd4180656b0 2024-04-16T07:20:18.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.731+0000 7fd415600700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 0x7fd40006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:18.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.731+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd40c088490 con 0x7fd4180656b0 2024-04-16T07:20:18.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.731+0000 7fd41ca00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 0x7fd40006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:18.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.732+0000 7fd41ca00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 0x7fd40006b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd418111600 tx=0x7fd404009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:18.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:18.736+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd40c058200 con 0x7fd4180656b0 2024-04-16T07:20:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:18 smithi104 ceph-mon[36163]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T07:20:19.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.003+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd41805f620 con 0x7fd4180656b0 2024-04-16T07:20:19.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.004+0000 7fd415600700 1 -- 172.21.15.26:0/3991147333 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fd40c057d90 con 0x7fd4180656b0 2024-04-16T07:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.007+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 msgr2=0x7fd40006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.007+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 0x7fd40006b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd418111600 tx=0x7fd404009ee0 comp rx=0 tx=0).stop 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.007+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 msgr2=0x7fd418115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.007+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009150 tx=0x7fd40c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 shutdown_connections 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fd4000692b0 0x7fd40006b760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4180656b0 0x7fd418115d60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 --2- 172.21.15.26:0/3991147333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd418102e60 0x7fd4181162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 >> 172.21.15.26:0/3991147333 conn(0x7fd418072bc0 msgr2=0x7fd4180fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 shutdown_connections 2024-04-16T07:20:19.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19.008+0000 7fd41fe00700 1 -- 172.21.15.26:0/3991147333 wait complete. 2024-04-16T07:20:19.734 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:19 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3991147333' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi026.front.sepia.ceph.com 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T07:20:19.784 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T07:20:19.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:19.784 DEBUG:teuthology.orchestra.run.smithi026:> ip addr 2024-04-16T07:20:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:19 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3991147333' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T07:20:19.843 INFO:teuthology.orchestra.run.smithi026.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 ::1/128 scope host 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6a brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f0 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6b brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f1 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c0 brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f0 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: inet 172.21.15.26/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft 41985sec preferred_lft 41985sec 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c1 brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f1 2024-04-16T07:20:19.846 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T07:20:19.846 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:19.846 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-16T07:20:19.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add name ceph-brx type bridge 2024-04-16T07:20:19.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr flush dev ceph-brx 2024-04-16T07:20:19.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set ceph-brx up 2024-04-16T07:20:19.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T07:20:19.847 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-16T07:20:19.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:20.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:20.395 DEBUG:teuthology.orchestra.run.smithi026:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T07:20:20.473 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-04-16T07:20:20.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:20.477 DEBUG:teuthology.orchestra.run.smithi026:> route 2024-04-16T07:20:20.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:20 smithi026 ceph-mon[28805]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:20.540 INFO:teuthology.orchestra.run.smithi026.stdout:Kernel IP routing table 2024-04-16T07:20:20.540 INFO:teuthology.orchestra.run.smithi026.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T07:20:20.540 INFO:teuthology.orchestra.run.smithi026.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T07:20:20.540 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T07:20:20.541 INFO:teuthology.orchestra.run.smithi026.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T07:20:20.542 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-16T07:20:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:20 smithi104 ceph-mon[36163]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:21.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:21.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:21.453 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-04-16T07:20:21.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:21.514 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list-id 2024-04-16T07:20:21.565 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:21.565 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-16T07:20:21.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:21.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-16T07:20:21.565 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-16T07:20:21.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:21.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:21.785 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-16T07:20:21.785 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-16T07:20:21.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:22.143 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:22.143 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-16T07:20:22.143 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set brx.0 up 2024-04-16T07:20:22.143 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T07:20:22.143 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-16T07:20:22.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:22.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:22.399 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-16T07:20:22.399 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T07:20:22.399 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:22.443 INFO:teuthology.orchestra.run.smithi026.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T07:20:22.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T07:20:22.443 DEBUG:teuthology.orchestra.run.smithi026:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:22.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo modprobe fuse 2024-04-16T07:20:22.615 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:22 smithi026 ceph-mon[28805]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:22.628 DEBUG:teuthology.orchestra.run.smithi026:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/proc 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/sys 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/dev 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/security 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/shm 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/pts 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/run 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/cgroup 2024-04-16T07:20:22.686 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/pstore 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/bpf 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/ 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/selinux 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/mqueue 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/hugepages 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/debug 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/tracing 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/config 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/1000 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/ceph 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/1f748ab781e32f7f6994665847abb9f12299027bd674cfe57b16b2ac60bcd67b/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/9b6dc7f271e510587bde7480e6a21a41030c62a11f144e8d863e7859cccd50b4/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/0 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/194930a8fe66e410dddad18eefbcc5aae3e6ad19dfe826809e927ea26efe14e6/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/8c70c92593cc7bf5737738de79d7d8d9e821b1aa7340f8f56e22f79d082ef2ea/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f9b8eda6665264809444c77c1895aa1b1c0106182e1bb071b396f94bd38c7412/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/bdb1348f12790c6fcbec46174cd9ca7e0cac15be696a6dbb3bc8fd93172d82c7/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/e8cd5096e4b1a6596177136061ea2dd75784eba0f6f6dadb074e1839ac7b1250/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f54540ea48a97f63cff28713d2ee3e49c5f99193f6e84410ecabcaf52e68a1ca/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/6a065d2b25de848aeccababa39b5886134e19fc6c857553859c99bee6828fd66/merged 2024-04-16T07:20:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/659dc7c267c9cc0cf9875d47a705f6689dab18922142ed76e948a99acd9add9c/merged 2024-04-16T07:20:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/516ddbf8075e155c7fc5db0176a25a0a72de7814152b10b930b983b45d5a1471/merged 2024-04-16T07:20:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/09e4e651ab9c733508afa90282264d429b60a22a63cb675e7c44022b460cfebd/merged 2024-04-16T07:20:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns 2024-04-16T07:20:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:22.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:22.690 DEBUG:teuthology.orchestra.run.smithi026:> ls /sys/fs/fuse/connections 2024-04-16T07:20:22.744 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T07:20:22.744 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f -d --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-16T07:20:22.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo modprobe fuse 2024-04-16T07:20:22.832 DEBUG:teuthology.orchestra.run.smithi026:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:22 smithi104 ceph-mon[36163]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:22.880 INFO:teuthology.orchestra.run.smithi026.stdout:/proc 2024-04-16T07:20:22.880 INFO:teuthology.orchestra.run.smithi026.stdout:/sys 2024-04-16T07:20:22.880 INFO:teuthology.orchestra.run.smithi026.stdout:/dev 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/security 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/shm 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/pts 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/run 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/cgroup 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/pstore 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/bpf 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/ 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/selinux 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/mqueue 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/hugepages 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/debug 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/tracing 2024-04-16T07:20:22.881 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/config 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/1000 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/ceph 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/1f748ab781e32f7f6994665847abb9f12299027bd674cfe57b16b2ac60bcd67b/merged 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/9b6dc7f271e510587bde7480e6a21a41030c62a11f144e8d863e7859cccd50b4/merged 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/0 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/194930a8fe66e410dddad18eefbcc5aae3e6ad19dfe826809e927ea26efe14e6/merged 2024-04-16T07:20:22.882 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/8c70c92593cc7bf5737738de79d7d8d9e821b1aa7340f8f56e22f79d082ef2ea/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f9b8eda6665264809444c77c1895aa1b1c0106182e1bb071b396f94bd38c7412/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/bdb1348f12790c6fcbec46174cd9ca7e0cac15be696a6dbb3bc8fd93172d82c7/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/e8cd5096e4b1a6596177136061ea2dd75784eba0f6f6dadb074e1839ac7b1250/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f54540ea48a97f63cff28713d2ee3e49c5f99193f6e84410ecabcaf52e68a1ca/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/6a065d2b25de848aeccababa39b5886134e19fc6c857553859c99bee6828fd66/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/659dc7c267c9cc0cf9875d47a705f6689dab18922142ed76e948a99acd9add9c/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/516ddbf8075e155c7fc5db0176a25a0a72de7814152b10b930b983b45d5a1471/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/09e4e651ab9c733508afa90282264d429b60a22a63cb675e7c44022b460cfebd/merged 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:22.883 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:22.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:22.884 DEBUG:teuthology.orchestra.run.smithi026:> ls /sys/fs/fuse/connections 2024-04-16T07:20:22.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 -- 172.21.15.26:0/665983049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa019a0 msgr2=0x563ccaa024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:22.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 --2- 172.21.15.26:0/665983049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa019a0 0x563ccaa024a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbf80004e50 tx=0x7fbf8001c8b0 comp rx=0 tx=0).stop 2024-04-16T07:20:22.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 -- 172.21.15.26:0/665983049 shutdown_connections 2024-04-16T07:20:22.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 --2- 172.21.15.26:0/665983049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa019a0 0x563ccaa024a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:22.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 --2- 172.21.15.26:0/665983049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x563ccaa01460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:22.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.966+0000 7fbf94c4e300 1 -- 172.21.15.26:0/665983049 >> 172.21.15.26:0/665983049 conn(0x563cca969fe0 msgr2=0x563cca96a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:22.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.967+0000 7fbf94c4e300 1 -- 172.21.15.26:0/665983049 shutdown_connections 2024-04-16T07:20:22.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.967+0000 7fbf94c4e300 1 -- 172.21.15.26:0/665983049 wait complete. 2024-04-16T07:20:22.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.967+0000 7fbf94c4e300 0 ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-fuse, pid 75906 2024-04-16T07:20:22.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.968+0000 7fbf94c4e300 -1 init, newargv = 0x563ccaa112b0 newargc=16 2024-04-16T07:20:22.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.968+0000 7fbf94c4e300 10 client.0 _ll_register_callbacks cb 0x563cca968530 invalidate_ino_cb 1 invalidate_dentry_cb 1 switch_interrupt_cb 1 remount_cb 1 2024-04-16T07:20:22.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:ceph-fuse[75906]: starting ceph client 2024-04-16T07:20:22.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.969+0000 7fbf94c4e300 1 -- start start 2024-04-16T07:20:22.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.969+0000 7fbf94c4e300 1 Processor -- start 2024-04-16T07:20:22.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf94c4e300 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x563ccaa155e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf94c4e300 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf94c4e300 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cca8d0da0 con 0x563ccaa15bb0 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf94c4e300 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cca8d0a40 con 0x563ccaa01090 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:192.168.0.1:57248/0 (socket says 192.168.0.1:57248) 2024-04-16T07:20:22.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 learned_addr learned my addr 192.168.0.1:0/2108819864 (peer_addr_for_me v2:192.168.0.1:0/0) 2024-04-16T07:20:22.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 msgr2=0x563ccaa155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:22.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x563ccaa155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:22.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.970+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf80005040 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf8001b420 tx=0x7fbf8001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf80058070 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbf80053470 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf94c4e300 10 client.14524 Subscribing to map 'mdsmap' 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,mdsmap=0+,monmap=3+}) v3 -- 0x563cca8f6d60 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 3 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf80051e80 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mdsmap(e 15) v2 ==== 1108+0+0 (secure 0 0 0) 0x7fbf80067080 con 0x563ccaa15bb0 2024-04-16T07:20:22.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.971+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 15 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 20 client.14524 populate_metadata read hostname 'smithi026' 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 10 client.14524 did not get mds through better means, so chose random mds 0 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 20 client.14524 mds is 0 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 10 client.14524 _open_mds_session mds.0 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 0x563ccaa1f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_open) v5 -- 0x563cca930030 con 0x563ccaa1cfa0 2024-04-16T07:20:22.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf94c4e300 10 client.14524 waiting for session to mds.0 to open 2024-04-16T07:20:22.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.972+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 0x563ccaa1f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:22.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.973+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 0x563ccaa1f380 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b62cd6c19ae5505 server_cookie=8de965e5f49e8398 in_seq=0 out_seq=0 2024-04-16T07:20:22.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.973+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.26:6828/1284848687 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x563cca930030 con 0x563ccaa1cfa0 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 renew_caps mds.0 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 1) v5 -- 0x7fbf80053470 con 0x563ccaa1cfa0 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:20:22.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 10 client.14524 did not get mds through better means, so chose random mds 0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 mds is 0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 10 client.14524 send_request rebuilding request 1 for mds.0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 encode_cap_releases enter (req: 0x563ccaa1cbc0, mds: 0) 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 send_request set sent_stamp to 2024-04-16T07:20:22.980273+0000 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 10 client.14524 send_request client_request(unknown.0:1 getattr pAsLsXsFs #0x1 2024-04-16T07:20:22.973056+0000 caller_uid=0, caller_gid=0{}) v5 to mds.0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_request(unknown.0:1 getattr pAsLsXsFs #0x1 2024-04-16T07:20:22.973056+0000 caller_uid=0, caller_gid=0{}) v5 -- 0x563ccaa1ff80 con 0x563ccaa1cfa0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 awaiting reply|forward|kick on 0x7ffce0388800 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 2 ==== client_session(renewcaps seq 1) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf80053470 con 0x563ccaa1cfa0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 1) v1 from mds.0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 3 ==== client_reply(???:1 = 0 (0) Success) v1 ==== 423+0+0 (crc 0 0 0) 0x7fbf8400a800 con 0x563ccaa1cfa0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 handle_client_reply got a reply. Safe:1 tid 1 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 insert_trace from 2024-04-16T07:20:22.980273+0000 mds.0 is_target=1 is_dentry=0 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 features 0xffffffffffffffff 2024-04-16T07:20:22.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0, nref was 0 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref now is 1 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x1 nref=1 c=0 seq=0 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) seq 1 > 0 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 invalidate_snaprealm_and_children snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 15 client.14524 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) self|parent updated 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 15 client.14524 snapc 1=[] 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 no new snap on snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 put_snap_realm 0x1 0x7fbf70007a40 2 -> 1 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 hrm is_target=1 is_dentry=0 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 update_inode_file_time 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=0.000000 ctime=0.000000 change_attr=0 caps=- 0x7fbf70007b30) - ctime 2024-04-16T07:19:43.615094+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 dir hash is 2 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 12 client.14524 add_update_inode client inode change_attr: 0 , mds inodestat change_attr: 0 2024-04-16T07:20:22.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 12 client.14524 add_update_inode adding 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=- has_dir_layout 0x7fbf70007b30) caps pAsLsXsFs 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref was 1 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref now is 2 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 15 client.14524 add_update_cap first one, opened snaprealm 0x7fbf70007a40 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 10 client.14524 marking (I_COMPLETE|I_DIR_ORDERED) on empty dir 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 put_snap_realm 0x1 0x7fbf70007a40 2 -> 1 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 handle_client_reply signalling caller 0x7ffce0388800 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 handle_client_reply awaiting kickback on tid 1 0x7fbf773fe000 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 sendrecv kickback on tid 1 0x7fbf773fe000 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 lat 0.000654 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf94c4e300 20 client.14524 _ll_get 0x7fbf70007b30 0x1 -> 1 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:22.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.979+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:22.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:FUSE library version: 2.9.9 2024-04-16T07:20:22.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:ceph-fuse[75906]: starting fuse 2024-04-16T07:20:22.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stdout:ceph_fuse: starting fuse loop 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.982+0000 7fbf94c4e300 -1 multithreaded fuse 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.982+0000 7fbf74c00640 10 client.14524 euid: 0 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:INIT: 7.40 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:flags=0x73fffffb 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:max_readahead=0x00020000 2024-04-16T07:20:22.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: INIT: 7.19 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: flags=0x0000043b 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: max_readahead=0x00020000 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: max_write=0x00020000 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.982+0000 7fbf74c00640 1 client.14524 using remount_cb 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: max_background=0 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: congestion_threshold=0 2024-04-16T07:20:22.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 2, success, outsize: 40 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75934 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 4, success, outsize: 120 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.988+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.988+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.988+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 040755 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.988+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:20:22.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:22.989+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:20:23.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 2) v5 -- 0x7fbf68001d80 con 0x563ccaa1cfa0 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf680025b0 con 0x563ccaa1cfa0 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 10 client.14524 _put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 2 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:23.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.972+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:23.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.973+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 4 ==== client_session(renewcaps seq 2) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf8400bbd0 con 0x563ccaa1cfa0 2024-04-16T07:20:23.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:23.973+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 2) v1 from mds.0 2024-04-16T07:20:24.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo modprobe fuse 2024-04-16T07:20:24.141 DEBUG:teuthology.orchestra.run.smithi026:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/proc 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/sys 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/dev 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/security 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/shm 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/pts 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/run 2024-04-16T07:20:24.200 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/cgroup 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/pstore 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/bpf 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/ 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/selinux 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/mqueue 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/hugepages 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/debug 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/tracing 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/sys/kernel/config 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:24.201 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/1000 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/ceph 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/1f748ab781e32f7f6994665847abb9f12299027bd674cfe57b16b2ac60bcd67b/merged 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/9b6dc7f271e510587bde7480e6a21a41030c62a11f144e8d863e7859cccd50b4/merged 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/run/user/0 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/194930a8fe66e410dddad18eefbcc5aae3e6ad19dfe826809e927ea26efe14e6/merged 2024-04-16T07:20:24.202 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/8c70c92593cc7bf5737738de79d7d8d9e821b1aa7340f8f56e22f79d082ef2ea/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f9b8eda6665264809444c77c1895aa1b1c0106182e1bb071b396f94bd38c7412/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/bdb1348f12790c6fcbec46174cd9ca7e0cac15be696a6dbb3bc8fd93172d82c7/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/e8cd5096e4b1a6596177136061ea2dd75784eba0f6f6dadb074e1839ac7b1250/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/f54540ea48a97f63cff28713d2ee3e49c5f99193f6e84410ecabcaf52e68a1ca/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/6a065d2b25de848aeccababa39b5886134e19fc6c857553859c99bee6828fd66/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/659dc7c267c9cc0cf9875d47a705f6689dab18922142ed76e948a99acd9add9c/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/516ddbf8075e155c7fc5db0176a25a0a72de7814152b10b930b983b45d5a1471/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/var/lib/containers/storage/overlay/09e4e651ab9c733508afa90282264d429b60a22a63cb675e7c44022b460cfebd/merged 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T07:20:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:24.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:24.205 DEBUG:teuthology.orchestra.run.smithi026:> ls /sys/fs/fuse/connections 2024-04-16T07:20:24.255 INFO:teuthology.orchestra.run.smithi026.stdout:148 2024-04-16T07:20:24.255 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [148] 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo stdin-killer -- python3 -c ' 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> import glob 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> import re 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> import os 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> import subprocess 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> def _find_admin_socket(client_name): 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> files = glob.glob(asok_path) 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> 2024-04-16T07:20:24.256 DEBUG:teuthology.orchestra.run.smithi026:> # Given a non-glob path, it better be there 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> if "*" not in asok_path: 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> assert(len(files) == 1) 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> return files[0] 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> for f in files: 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> contents = proc_f.read() 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> if mountpoint in contents: 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> return f 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> print(_find_admin_socket("client.0")) 2024-04-16T07:20:24.257 DEBUG:teuthology.orchestra.run.smithi026:> ' 2024-04-16T07:20:24.381 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:24 smithi026 ceph-mon[28805]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:24.437 INFO:teuthology.orchestra.run.smithi026.stdout:/var/run/ceph/ceph-client.0.75906.asok 2024-04-16T07:20:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:24.457 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75906.asok 2024-04-16T07:20:24.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:24.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75906.asok status 2024-04-16T07:20:24.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.690+0000 7fbf8be00640 1 client.14524 dump_status 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": { 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "entity_id": "0", 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "hostname": "smithi026", 2024-04-16T07:20:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "pid": "75906", 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "root": "/" 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "dentry_count": 0, 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "dentry_pinned_count": 0, 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 14524, 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "inst": { 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "name": { 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "client", 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 14524 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": { 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "192.168.0.1:0", 2024-04-16T07:20:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2108819864 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": { 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "192.168.0.1:0", 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2108819864 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "inst_str": "client.14524 192.168.0.1:0/2108819864", 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr_str": "192.168.0.1:0/2108819864", 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "inode_count": 1, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_epoch": 15, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_epoch": 47, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_epoch_barrier": 0, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklisted": false, 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs" 2024-04-16T07:20:24.691 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:20:24.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T07:20:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:24 smithi104 ceph-mon[36163]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf680029b0 con 0x563ccaa1cfa0 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:24.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:24.972+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68002d80 con 0x563ccaa1cfa0 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:25.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:25.972+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:26 smithi026 ceph-mon[28805]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:26 smithi104 ceph-mon[36163]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:26.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:26.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:26.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:26.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003120 con 0x563ccaa1cfa0 2024-04-16T07:20:26.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:26.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:26.972+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:27.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.248+0000 7fcd34400700 1 -- 172.21.15.26:0/1341117932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c1059c0 msgr2=0x7fcd2c0ff030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:27.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.248+0000 7fcd34400700 1 --2- 172.21.15.26:0/1341117932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c0ff030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcd18004940 tx=0x7fcd18009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:27.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 -- 172.21.15.26:0/1341117932 shutdown_connections 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 --2- 172.21.15.26:0/1341117932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c0ff030 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 --2- 172.21.15.26:0/1341117932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c105020 0x7fcd2c1053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 -- 172.21.15.26:0/1341117932 >> 172.21.15.26:0/1341117932 conn(0x7fcd2c0fabd0 msgr2=0x7fcd2c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 -- 172.21.15.26:0/1341117932 shutdown_connections 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.249+0000 7fcd34400700 1 -- 172.21.15.26:0/1341117932 wait complete. 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 Processor -- start 2024-04-16T07:20:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 -- start start 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c1160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c116730 con 0x7fcd2c105020 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.250+0000 7fcd34400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c10fd10 con 0x7fcd2c1059c0 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50830/0 (socket says 172.21.15.26:50830) 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 -- 172.21.15.26:0/1028554466 learned_addr learned my addr 172.21.15.26:0/1028554466 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31000700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c1160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 -- 172.21.15.26:0/1028554466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c1059c0 msgr2=0x7fcd2c1160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c1160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 -- 172.21.15.26:0/1028554466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18005040 con 0x7fcd2c105020 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.251+0000 7fcd31a00700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcd20002a50 tx=0x7fcd200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.252+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd20007cc0 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.252+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd2000f040 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.252+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c10fff0 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.252+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd20009640 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.252+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c1105b0 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.253+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c100770 con 0x7fcd2c105020 2024-04-16T07:20:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.254+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fcd200097a0 con 0x7fcd2c105020 2024-04-16T07:20:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.255+0000 7fcd29600700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 0x7fcd1406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.255+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcd200870c0 con 0x7fcd2c105020 2024-04-16T07:20:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.255+0000 7fcd31000700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 0x7fcd1406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.256+0000 7fcd31000700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 0x7fcd1406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c111440 tx=0x7fcd18009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:27.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.260+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fcd20056d10 con 0x7fcd2c105020 2024-04-16T07:20:27.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.526+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcd2c1111f0 con 0x7fcd2c105020 2024-04-16T07:20:27.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.526+0000 7fcd29600700 1 -- 172.21.15.26:0/1028554466 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fcd200568a0 con 0x7fcd2c105020 2024-04-16T07:20:27.526 INFO:teuthology.orchestra.run.smithi026.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T07:20:27.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 msgr2=0x7fcd1406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:27.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 0x7fcd1406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c111440 tx=0x7fcd18009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:27.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 msgr2=0x7fcd2c115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:27.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcd20002a50 tx=0x7fcd200046b0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 shutdown_connections 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fcd140692a0 0x7fcd1406b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd2c105020 0x7fcd2c115ba0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 --2- 172.21.15.26:0/1028554466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c1059c0 0x7fcd2c1160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 >> 172.21.15.26:0/1028554466 conn(0x7fcd2c0fabd0 msgr2=0x7fcd2c0fbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.529+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 shutdown_connections 2024-04-16T07:20:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:27.530+0000 7fcd34400700 1 -- 172.21.15.26:0/1028554466 wait complete. 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf680034c0 con 0x563ccaa1cfa0 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:27.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:27.973+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:28.305 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T07:20:28.305 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T07:20:28.305 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-04-16T07:20:28.305 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T07:20:28.305 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:28.306 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T07:20:28.306 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T07:20:28.306 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T07:20:28.306 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T07:20:28.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:28.306 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-04-16T07:20:28.366 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-04-16T07:20:28.367 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41969sec preferred_lft 41969sec 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-04-16T07:20:28.368 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-04-16T07:20:28.368 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T07:20:28.368 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:28.368 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-04-16T07:20:28.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2024-04-16T07:20:28.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2024-04-16T07:20:28.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2024-04-16T07:20:28.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T07:20:28.369 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-04-16T07:20:28.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:28.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:28 smithi026 ceph-mon[28805]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:28.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:28 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1028554466' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T07:20:28.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:28 smithi104 ceph-mon[36163]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:28.737 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:28 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1028554466' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T07:20:28.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:28.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:28.881 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T07:20:28.954 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-04-16T07:20:28.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:28.958 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003860 con 0x563ccaa1cfa0 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:28.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:28.973+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-04-16T07:20:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T07:20:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T07:20:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T07:20:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T07:20:29.018 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-04-16T07:20:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:29.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:29.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:29.840 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-04-16T07:20:29.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:29.894 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2024-04-16T07:20:29.948 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:29.949 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-04-16T07:20:29.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:29.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-16T07:20:29.949 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:29.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:29.974+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:30.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:30.156 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-16T07:20:30.156 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:30.156 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-04-16T07:20:30.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-16T07:20:30.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T07:20:30.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-16T07:20:30.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-16T07:20:30.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-16T07:20:30.157 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-04-16T07:20:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:30.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:30.482 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T07:20:30.482 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-04-16T07:20:30.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-04-16T07:20:30.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T07:20:30.483 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-04-16T07:20:30.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T07:20:30.607 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:30 smithi104 ceph-mon[36163]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:30.704 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-16T07:20:30.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T07:20:30.704 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:30.750 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T07:20:30.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T07:20:30.750 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:30 smithi026 ceph-mon[28805]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:30.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-04-16T07:20:30.931 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:30.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:30.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:30.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:30.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:30.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:30.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:30.974+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:30.984 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-04-16T07:20:30.984 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-04-16T07:20:30.984 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-04-16T07:20:30.984 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/fec0aedfd80826bcb50fdbc60c8f720c6e8bc02e868652cd5f2e0d5ffefff3dd/merged 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/d0c9f7bd641cd685781f89934e717bf695ebf1d587d2fa42f675a730eb5d0bd4/merged 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f9f2a19987c2f9176b3a24498c532f144d10f2cc4cc70f30af18737d0f092975/merged 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/6bfe8c7b6b112db3ba9012c9dd97a016261d3be71c9bb37224de59b071664a77/merged 2024-04-16T07:20:30.986 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/244b7ca0701cbbed997edc0377c91d79569ad05b57b8362877e932f3825a2465/merged 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/4a5fc1032acaf2c429a80006c2aa2207118d206d82d8e6e4784d309c6fcc9235/merged 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/7267620836ed8f7f4a5d08a5d8c0730139809a1ceace40496552b5084a4d9071/merged 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f18dafa11c4973618ce1340061e1d5806f1af19b35a2b250a5db220ad4dd6066/merged 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/aa698c0f8ad58878eb9263def54cc542014b81fe09f8d8f28a26bb6b93c9f263/merged 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:30.987 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:30.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:30.988 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-04-16T07:20:31.045 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T07:20:31.046 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f -d --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-16T07:20:31.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-04-16T07:20:31.127 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:31.171 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-04-16T07:20:31.171 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-04-16T07:20:31.171 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-04-16T07:20:31.171 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-04-16T07:20:31.172 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/fec0aedfd80826bcb50fdbc60c8f720c6e8bc02e868652cd5f2e0d5ffefff3dd/merged 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/d0c9f7bd641cd685781f89934e717bf695ebf1d587d2fa42f675a730eb5d0bd4/merged 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f9f2a19987c2f9176b3a24498c532f144d10f2cc4cc70f30af18737d0f092975/merged 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/6bfe8c7b6b112db3ba9012c9dd97a016261d3be71c9bb37224de59b071664a77/merged 2024-04-16T07:20:31.173 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/244b7ca0701cbbed997edc0377c91d79569ad05b57b8362877e932f3825a2465/merged 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/4a5fc1032acaf2c429a80006c2aa2207118d206d82d8e6e4784d309c6fcc9235/merged 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/7267620836ed8f7f4a5d08a5d8c0730139809a1ceace40496552b5084a4d9071/merged 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f18dafa11c4973618ce1340061e1d5806f1af19b35a2b250a5db220ad4dd6066/merged 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/aa698c0f8ad58878eb9263def54cc542014b81fe09f8d8f28a26bb6b93c9f263/merged 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:31.174 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:31.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:31.175 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/4087956397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x55fc3f5014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 --2- 192.168.0.1:0/4087956397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x55fc3f5014a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0004f10 tx=0x7fe9c001c970 comp rx=0 tx=0).stop 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/4087956397 shutdown_connections 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 --2- 192.168.0.1:0/4087956397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x55fc3f5014a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 --2- 192.168.0.1:0/4087956397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f500090 0x55fc3f500460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/4087956397 >> 192.168.0.1:0/4087956397 conn(0x55fc3f468fe0 msgr2=0x55fc3f4693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/4087956397 shutdown_connections 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/4087956397 wait complete. 2024-04-16T07:20:31.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.259+0000 7fe9d8cc2300 0 ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-fuse, pid 60964 2024-04-16T07:20:31.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.261+0000 7fe9d8cc2300 -1 init, newargv = 0x55fc3f5100f0 newargc=16 2024-04-16T07:20:31.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.261+0000 7fe9d8cc2300 10 client.0 _ll_register_callbacks cb 0x55fc3f3ca2b0 invalidate_ino_cb 1 invalidate_dentry_cb 1 switch_interrupt_cb 1 remount_cb 1 2024-04-16T07:20:31.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:ceph-fuse[60964]: starting ceph client 2024-04-16T07:20:31.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.261+0000 7fe9d8cc2300 1 -- start start 2024-04-16T07:20:31.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.261+0000 7fe9d8cc2300 1 Processor -- start 2024-04-16T07:20:31.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d8cc2300 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:31.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d8cc2300 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x55fc3f514a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:31.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d8cc2300 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc3f3cfda0 con 0x55fc3f5009a0 2024-04-16T07:20:31.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d8cc2300 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc3f3cfa40 con 0x55fc3f500090 2024-04-16T07:20:31.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d4a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:31.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d4a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:192.168.0.1:45154/0 (socket says 192.168.0.1:45154) 2024-04-16T07:20:31.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.262+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 learned_addr learned my addr 192.168.0.1:0/874387268 (peer_addr_for_me v2:192.168.0.1:0/0) 2024-04-16T07:20:31.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x55fc3f514a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:31.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 msgr2=0x55fc3f514a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x55fc3f514a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0005040 con 0x55fc3f500090 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800afa0 tx=0x7fe9c8004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9baa00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.263+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c800f070 con 0x55fc3f500090 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 10 client.24339 Subscribing to map 'mdsmap' 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mdsmap=0+,monmap=3+}) v3 -- 0x55fc3f3f5d60 con 0x55fc3f500090 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe9c8006560 con 0x55fc3f500090 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:31.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 3 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9c8006bf0 con 0x55fc3f500090 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mdsmap(e 15) v2 ==== 1108+0+0 (secure 0 0 0) 0x7fe9c801e080 con 0x55fc3f500090 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 15 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 20 client.24339 populate_metadata read hostname 'smithi104' 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 10 client.24339 did not get mds through better means, so chose random mds 0 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 20 client.24339 mds is 0 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 10 client.24339 _open_mds_session mds.0 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 0x55fc3f51de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_open) v5 -- 0x55fc3f42f030 con 0x55fc3f51baa0 2024-04-16T07:20:31.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.264+0000 7fe9d8cc2300 10 client.24339 waiting for session to mds.0 to open 2024-04-16T07:20:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.265+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 0x55fc3f51de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.266+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 0x55fc3f51de80 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d3075d49c878d6 server_cookie=d4df63255852442f in_seq=0 out_seq=0 2024-04-16T07:20:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.266+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.26:6828/1284848687 2024-04-16T07:20:31.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x55fc3f42f030 con 0x55fc3f51baa0 2024-04-16T07:20:31.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:20:31.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 10 client.24339 renew_caps mds.0 2024-04-16T07:20:31.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 1) v5 -- 0x7fe9c8006560 con 0x55fc3f51baa0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 10 client.24339 did not get mds through better means, so chose random mds 0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 20 client.24339 mds is 0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 10 client.24339 send_request rebuilding request 1 for mds.0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 20 client.24339 encode_cap_releases enter (req: 0x55fc3f51b690, mds: 0) 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 20 client.24339 send_request set sent_stamp to 2024-04-16T07:20:31.270998+0000 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 10 client.24339 send_request client_request(unknown.0:1 getattr pAsLsXsFs #0x1 2024-04-16T07:20:31.265342+0000 caller_uid=0, caller_gid=0{}) v5 to mds.0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_request(unknown.0:1 getattr pAsLsXsFs #0x1 2024-04-16T07:20:31.265342+0000 caller_uid=0, caller_gid=0{}) v5 -- 0x55fc3f51ea10 con 0x55fc3f51baa0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9d8cc2300 20 client.24339 awaiting reply|forward|kick on 0x7fff23cd97a0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 2 ==== client_session(renewcaps seq 1) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8006560 con 0x55fc3f51baa0 2024-04-16T07:20:31.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.270+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 1) v1 from mds.0 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 3 ==== client_reply(???:1 = 0 (0) Success) v1 ==== 423+0+0 (crc 0 0 0) 0x7fe9c004e070 con 0x55fc3f51baa0 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 handle_client_reply got a reply. Safe:1 tid 1 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 insert_trace from 2024-04-16T07:20:31.270998+0000 mds.0 is_target=1 is_dentry=0 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 features 0xffffffffffffffff 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 update_snap_trace len 70 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0, nref was 0 2024-04-16T07:20:31.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0x7fe9b4007a40, nref now is 1 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 update_snap_trace snaprealm(0x1 nref=1 c=0 seq=0 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) seq 1 > 0 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 invalidate_snaprealm_and_children snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 15 client.24339 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) self|parent updated 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 15 client.24339 snapc 1=[] 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 no new snap on snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x1 0x7fe9b4007a40 2 -> 1 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 hrm is_target=1 is_dentry=0 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 update_inode_file_time 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=0.000000 ctime=0.000000 change_attr=0 caps=- 0x7fe9b4007b30) - ctime 2024-04-16T07:19:43.615094+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 dir hash is 2 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 12 client.24339 add_update_inode client inode change_attr: 0 , mds inodestat change_attr: 0 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 12 client.24339 add_update_inode adding 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=- has_dir_layout 0x7fe9b4007b30) caps pAsLsXsFs 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0x7fe9b4007a40, nref was 1 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0x7fe9b4007a40, nref now is 2 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 15 client.24339 add_update_cap first one, opened snaprealm 0x7fe9b4007a40 2024-04-16T07:20:31.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 10 client.24339 marking (I_COMPLETE|I_DIR_ORDERED) on empty dir 0x1.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x1 0x7fe9b4007a40 2 -> 1 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 handle_client_reply signalling caller 0x7fff23cd97a0 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 handle_client_reply awaiting kickback on tid 1 0x7fe9ba9fe000 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9d8cc2300 20 client.24339 sendrecv kickback on tid 1 0x7fe9ba9fe000 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9d8cc2300 20 client.24339 lat 0.001184 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9d8cc2300 20 client.24339 _ll_get 0x7fe9b4007b30 0x1 -> 1 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.271+0000 7fe9baa00640 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:31.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:FUSE library version: 2.9.9 2024-04-16T07:20:31.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:ceph-fuse[60964]: starting fuse 2024-04-16T07:20:31.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stdout:ceph_fuse: starting fuse loop 2024-04-16T07:20:31.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.273+0000 7fe9d8cc2300 -1 multithreaded fuse 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.273+0000 7fe9b3e00640 10 client.24339 euid: 0 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.273+0000 7fe9b3e00640 1 client.24339 using remount_cb 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:INIT: 7.40 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:flags=0x73fffffb 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:max_readahead=0x00020000 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: INIT: 7.19 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: flags=0x0000043b 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: max_readahead=0x00020000 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: max_write=0x00020000 2024-04-16T07:20:31.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: max_background=0 2024-04-16T07:20:31.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: congestion_threshold=0 2024-04-16T07:20:31.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 2, success, outsize: 40 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61008 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.278+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 4, success, outsize: 120 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.278+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.278+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 040755 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.278+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:20:31.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:31.278+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:20:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:31.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:31.974+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 2) v5 -- 0x7fe9ac001d80 con 0x55fc3f51baa0 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.001184, avg_latency: 0.001184, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0025b0 con 0x55fc3f51baa0 2024-04-16T07:20:32.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 10 client.24339 _put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 2 2024-04-16T07:20:32.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:32.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.264+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:32.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.265+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 4 ==== client_session(renewcaps seq 2) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c00513c0 con 0x55fc3f51baa0 2024-04-16T07:20:32.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:32.265+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 2) v1 from mds.0 2024-04-16T07:20:32.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-04-16T07:20:32.471 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-04-16T07:20:32.527 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/fec0aedfd80826bcb50fdbc60c8f720c6e8bc02e868652cd5f2e0d5ffefff3dd/merged 2024-04-16T07:20:32.528 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/d0c9f7bd641cd685781f89934e717bf695ebf1d587d2fa42f675a730eb5d0bd4/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f9f2a19987c2f9176b3a24498c532f144d10f2cc4cc70f30af18737d0f092975/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/6bfe8c7b6b112db3ba9012c9dd97a016261d3be71c9bb37224de59b071664a77/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/244b7ca0701cbbed997edc0377c91d79569ad05b57b8362877e932f3825a2465/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/4a5fc1032acaf2c429a80006c2aa2207118d206d82d8e6e4784d309c6fcc9235/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/7267620836ed8f7f4a5d08a5d8c0730139809a1ceace40496552b5084a4d9071/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f18dafa11c4973618ce1340061e1d5806f1af19b35a2b250a5db220ad4dd6066/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/aa698c0f8ad58878eb9263def54cc542014b81fe09f8d8f28a26bb6b93c9f263/merged 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T07:20:32.529 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:32.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:32.530 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-04-16T07:20:32.580 INFO:teuthology.orchestra.run.smithi104.stdout:115 2024-04-16T07:20:32.581 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo stdin-killer -- python3 -c ' 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> import re 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> import os 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2024-04-16T07:20:32.581 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> return f 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T07:20:32.582 DEBUG:teuthology.orchestra.run.smithi104:> 2024-04-16T07:20:32.583 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.1")) 2024-04-16T07:20:32.583 DEBUG:teuthology.orchestra.run.smithi104:> ' 2024-04-16T07:20:32.665 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:32 smithi104 ceph-mon[36163]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:32.757 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.1.60964.asok 2024-04-16T07:20:32.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:32 smithi026 ceph-mon[28805]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:32.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-16T07:20:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T07:20:32.774 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.60964.asok 2024-04-16T07:20:32.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:32.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.60964.asok status 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.000654, avg_latency: 0.000654, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:32.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:32.974+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:33.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.006+0000 7fe9cf400640 1 client.24339 dump_status 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "1", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "60964", 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24339, 2024-04-16T07:20:33.005 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 24339 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 874387268 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-04-16T07:20:33.006 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 874387268 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.24339 192.168.0.1:0/874387268", 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/874387268", 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 15, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 47, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2024-04-16T07:20:33.007 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-16T07:20:33.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:33.019 DEBUG:teuthology.orchestra.run.smithi026:> stat --file-system '--printf=%T 2024-04-16T07:20:33.019 DEBUG:teuthology.orchestra.run.smithi026:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:33.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 76196 2024-04-16T07:20:33.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.084+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- statfs(1 pool 3 v0) v2 -- 0x563cca923d30 con 0x563ccaa15bb0 2024-04-16T07:20:33.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.084+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 5 ==== statfs_reply(1) v1 ==== 56+0+0 (secure 0 0 0) 0x563cca923d30 con 0x563ccaa15bb0 2024-04-16T07:20:33.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.085+0000 7fbf94c4e300 10 client.14524 get_quota_root realm 0x1 2024-04-16T07:20:33.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.085+0000 7fbf94c4e300 10 client.14524 get_quota_root 0x1.head -> 0x1.head 2024-04-16T07:20:33.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 6, success, outsize: 96 2024-04-16T07:20:33.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.085+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=5 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:33.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.085+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:20:33.084 INFO:teuthology.orchestra.run.smithi026.stdout:fuseblk 2024-04-16T07:20:33.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:33.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:33.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T07:20:33.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76229 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 8, success, outsize: 120 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.190+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.190+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.190+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 040755 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.190+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:20:33.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.190+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 76229 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 8 client.14524 _ll_setattrx 0x1.head mask 1 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 may_setattr 0x1.head(faked_ino=0 nref=6 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30); UserPerm(uid: 0, gid: 0) stx_mode: 43ff mask:1 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 _getattr mask As issued=1 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 may_setattr 0x1.head(faked_ino=0 nref=6 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) = 0 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 3 client.14524 may_setattr 0x7fbf70007b30 = 0 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 _do_setattr mask 1 issued pAsLsXsFs 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 changing mode to 17407 2024-04-16T07:20:33.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 choose_target_mds starting with req->inode 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 choose_target_mds 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) is_hash=0 hash=0 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 choose_target_mds from caps on inode 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 mds is 0 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 send_request rebuilding request 2 for mds.0 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 encode_cap_releases enter (req: 0x563ccaa42bc0, mds: 0) 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 encode_inode_release enter(in:0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30), req:0x563ccaa42bc0 mds:0, drop:As, unless:-, force:0) 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 send_request set sent_stamp to 2024-04-16T07:20:33.192593+0000 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 10 client.14524 send_request client_request(unknown.0:2 setattr mode=041777 #0x1 2024-04-16T07:20:33.192545+0000 caller_uid=0, caller_gid=0{0,}) v5 to mds.0 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_request(unknown.0:2 setattr mode=041777 #0x1 2024-04-16T07:20:33.192545+0000 caller_uid=0, caller_gid=0{0,}) v5 -- 0x563ccaa42ec0 con 0x563ccaa1cfa0 2024-04-16T07:20:33.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.191+0000 7fbf94c4e300 20 client.14524 awaiting reply|forward|kick on 0x7ffce0388230 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.192+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 5 ==== client_caps(revoke ino 0x1 2 seq 2 caps=pLsXsFs dirty=- wanted=- follows 0 size 0/0 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:19:43.615094+0000 change_attr 0) v11 ==== 252+0+0 (crc 0 0 0) 0x7fe9c0053470 con 0x55fc3f51baa0 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 5 client.24339 set_cap_epoch_barrier epoch = 47 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 mds.0 seq now 1 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 5 client.24339 handle_cap_grant on in 0x1 mds.0 seq 2 caps now pLsXsFs was pAsLsXsFs 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 update_inode_file_time 0x1.head(faked_ino=0 nref=4 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) pAsLsXsFs ctime 2024-04-16T07:19:43.615094+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 revocation of As 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 check_caps on 0x1.head(faked_ino=0 nref=4 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) wanted AsLsXsFsx used - issued pLsXsFs revoking As flags=1 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 cap mds.0 issued pLsXsFs implemented pAsLsXsFs revoking As 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 completed revocation of As 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 10 client.24339 send_cap 0x1.head(faked_ino=0 nref=4 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) mds.0 seq 2 used - want AsLsXsFsx flush - retain pLsXsFsx held pAsLsXsFs revoking As dropping - 2024-04-16T07:20:33.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 15 client.24339 auth cap, requesting max_size 0 2024-04-16T07:20:33.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.193+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_caps(update ino 0x1 2 seq 2 caps=pLsXsFs dirty=- wanted=AsLsXsFsx follows 0 size 0/0 ts 1/18446744073709551615 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:19:43.615094+0000 change_attr 0) v11 -- 0x7fe9b40081f0 con 0x55fc3f51baa0 2024-04-16T07:20:33.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 5 ==== client_reply(???:2 = 0 (0) Success unsafe) v1 ==== 419+0+0 (crc 0 0 0) 0x7fbf8400be10 con 0x563ccaa1cfa0 2024-04-16T07:20:33.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 handle_client_reply got a reply. Safe:0 tid 2 2024-04-16T07:20:33.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 insert_trace from 2024-04-16T07:20:33.192593+0000 mds.0 is_target=1 is_dentry=0 2024-04-16T07:20:33.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 features 0xffffffffffffffff 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref was 1 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref now is 2 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 hrm is_target=1 is_dentry=0 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 12 client.14524 add_update_inode had 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) caps pLsXsFs 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 update_inode_file_time 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) pLsXsFs ctime 2024-04-16T07:20:33.192545+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 dir hash is 2 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 12 client.14524 add_update_inode client inode change_attr: 0 , mds inodestat change_attr: 1 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 add_update_cap issued pLsXsFs -> pLsXsFs from mds.0 on 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 10 client.14524 marking (I_COMPLETE|I_DIR_ORDERED) on empty dir 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 put_snap_realm 0x1 0x7fbf70007a40 2 -> 1 2024-04-16T07:20:33.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 handle_client_reply signalling caller 0x7ffce0388230 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf77400640 20 client.14524 handle_client_reply awaiting kickback on tid 2 0x7fbf773fe000 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 20 client.14524 sendrecv kickback on tid 2 0x7fbf773fe000 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 20 client.14524 make_request target is 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 20 client.14524 lat 0.002359 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 10, success, outsize: 120 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 10 client.14524 _setattr result=0 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 3 client.14524 ll_setattr 0x1.head = 0 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.193+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.194+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:20:33.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:33.198 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2024-04-16T07:20:33.198 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 61099 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.258+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- statfs(1 pool 3 v0) v2 -- 0x55fc3f422d30 con 0x55fc3f500090 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.259+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 5 ==== statfs_reply(1) v1 ==== 56+0+0 (secure 0 0 0) 0x55fc3f422d30 con 0x55fc3f500090 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.259+0000 7fe9d8cc2300 10 client.24339 get_quota_root realm 0x1 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 6, success, outsize: 96 2024-04-16T07:20:33.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.259+0000 7fe9d8cc2300 10 client.24339 get_quota_root 0x1.head -> 0x1.head 2024-04-16T07:20:33.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.259+0000 7fe9d8cc2300 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=5 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:33.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.259+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:20:33.259 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2024-04-16T07:20:33.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:33.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:20:33.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.001184, avg_latency: 0.001184, sq_sum: 0, count=1][client_metric_type: CAP_INFO cap_hits: 1 cap_misses: 0 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac002930 con 0x55fc3f51baa0 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 10 client.24339 _put_inode on 0x1.head(faked_ino=0 nref=5 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.264+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:33.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61132 2024-04-16T07:20:33.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:20:33.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=0 2024-04-16T07:20:33.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 choose_target_mds starting with req->inode 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 choose_target_mds 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) is_hash=0 hash=0 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 10 client.24339 choose_target_mds from caps on inode 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 mds is 0 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 10 client.24339 send_request rebuilding request 2 for mds.0 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 encode_cap_releases enter (req: 0x55fc3f541600, mds: 0) 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 send_request set sent_stamp to 2024-04-16T07:20:33.355198+0000 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 10 client.24339 send_request client_request(unknown.0:2 getattr pAsLsXsFs #0x1 2024-04-16T07:20:33.355173+0000 caller_uid=0, caller_gid=0{0,}) v5 to mds.0 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_request(unknown.0:2 getattr pAsLsXsFs #0x1 2024-04-16T07:20:33.355173+0000 caller_uid=0, caller_gid=0{0,}) v5 -- 0x55fc3f541900 con 0x55fc3f51baa0 2024-04-16T07:20:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.354+0000 7fe9d8cc2300 20 client.24339 awaiting reply|forward|kick on 0x7fff23cd9300 2024-04-16T07:20:33.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 6 ==== client_reply(???:2 = 0 (0) Success) v1 ==== 423+0+0 (crc 0 0 0) 0x7fe9c0051580 con 0x55fc3f51baa0 2024-04-16T07:20:33.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 handle_client_reply got a reply. Safe:1 tid 2 2024-04-16T07:20:33.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 insert_trace from 2024-04-16T07:20:33.355198+0000 mds.0 is_target=1 is_dentry=0 2024-04-16T07:20:33.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 features 0xffffffffffffffff 2024-04-16T07:20:33.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 update_snap_trace len 70 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0x7fe9b4007a40, nref was 1 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x1 0x7fe9b4007a40, nref now is 2 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 hrm is_target=1 is_dentry=0 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 12 client.24339 add_update_inode had 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=40755 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) caps pAsLsXsFs 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 update_inode_file_time 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:19:43.615094+0000 change_attr=0 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) pLsXsFs ctime 2024-04-16T07:20:33.192545+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 dir hash is 2 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 12 client.24339 add_update_inode client inode change_attr: 0 , mds inodestat change_attr: 1 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 add_update_cap issued pLsXsFs -> pAsLsXsFs from mds.0 on 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 10 client.24339 marking (I_COMPLETE|I_DIR_ORDERED) on empty dir 0x1.head(faked_ino=0 nref=5 ll_ref=4 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x1 0x7fe9b4007a40 2 -> 1 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 handle_client_reply signalling caller 0x7fff23cd9300 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 handle_client_reply awaiting kickback on tid 2 0x7fe9ba9fe000 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 20 client.24339 sendrecv kickback on tid 2 0x7fe9ba9fe000 2024-04-16T07:20:33.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 20 client.24339 lat 0.005604 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 8, success, outsize: 120 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 10 client.24339 _getattr result=0 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=4 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9baa00640 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=4 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.359+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:20:33.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 61132 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 6 ==== client_reply(???:2 = 0 (0) Success safe) v1 ==== 27+0+0 (crc 0 0 0) 0x7fbf84016660 con 0x563ccaa1cfa0 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 20 client.14524 handle_client_reply got a reply. Safe:1 tid 2 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 10 client.14524 insert_trace from 2024-04-16T07:20:33.192593+0000 mds.0 is_target=0 is_dentry=0 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 10 client.14524 insert_trace -- already got unsafe; ignoring 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:33.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.359+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 8 client.24339 _ll_setattrx 0x1.head mask 1 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 20 client.24339 may_setattr 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30); UserPerm(uid: 0, gid: 0) stx_mode: 43ff mask:1 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 10 client.24339 _getattr mask As issued=1 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 20 client.24339 may_setattr 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) = 0 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 3 client.24339 may_setattr 0x7fe9b4007b30 = 0 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 10 client.24339 _do_setattr mask 1 issued pAsLsXsFs 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 10 client.24339 changing mode to 17407 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 3 client.24339 ll_setattr 0x1.head = 0 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.360+0000 7fe9d8cc2300 20 client.24339 put_inode on 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=2 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 1 2024-04-16T07:20:33.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 10, success, outsize: 120 2024-04-16T07:20:33.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:33.361+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:20:33.363 INFO:teuthology.run_tasks:Running task print... 2024-04-16T07:20:33.373 INFO:teuthology.task.print:**** done client 2024-04-16T07:20:33.374 INFO:teuthology.run_tasks:Running task parallel... 2024-04-16T07:20:33.400 INFO:teuthology.task.parallel:starting parallel... 2024-04-16T07:20:33.400 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-16T07:20:33.414 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T07:20:33.414 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:20:33.414 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-16T07:20:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 10 client.14524 _put_inode on 0x1.head(faked_ino=0 nref=9 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 5 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:33.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:33.974+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:34.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.264+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:34.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:34.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.264+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:34.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.265+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac002cd0 con 0x55fc3f51baa0 2024-04-16T07:20:34.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.265+0000 7fe9b8c00640 10 client.24339 _put_inode on 0x1.head(faked_ino=0 nref=7 ll_ref=5 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=2 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) n = 3 2024-04-16T07:20:34.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.265+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:34.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:34.265+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:34 smithi026 ceph-mon[28805]: pgmap v246: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:34 smithi104 ceph-mon[36163]: pgmap v246: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:34.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.974+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:34.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.974+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:34.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.974+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:34.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.974+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:34.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.975+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:34.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:34.975+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:35.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:35.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:35.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:35.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003070 con 0x55fc3f51baa0 2024-04-16T07:20:35.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:35.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:35.265+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:35.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:35.975+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 -- 172.21.15.26:0/2760347592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 msgr2=0x7fcd080ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 --2- 172.21.15.26:0/2760347592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd080ff430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fccfc004940 tx=0x7fccfc009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 -- 172.21.15.26:0/2760347592 shutdown_connections 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 --2- 172.21.15.26:0/2760347592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 0x7fcd08100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 --2- 172.21.15.26:0/2760347592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd080ff430 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 -- 172.21.15.26:0/2760347592 >> 172.21.15.26:0/2760347592 conn(0x7fcd080fa950 msgr2=0x7fcd080fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 -- 172.21.15.26:0/2760347592 shutdown_connections 2024-04-16T07:20:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.132+0000 7fcd10200700 1 -- 172.21.15.26:0/2760347592 wait complete. 2024-04-16T07:20:36.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.133+0000 7fcd10200700 1 Processor -- start 2024-04-16T07:20:36.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd10200700 1 -- start start 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd10200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd0810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd10200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 0x7fcd081100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd10200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd081106a0 con 0x7fcd080ff020 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd10200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08110810 con 0x7fcd081002d0 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd0d800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd0810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.134+0000 7fcd0d800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd0810fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41900/0 (socket says 172.21.15.26:41900) 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0d800700 1 -- 172.21.15.26:0/2865187444 learned_addr learned my addr 172.21.15.26:0/2865187444 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0ce00700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 0x7fcd081100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0d800700 1 -- 172.21.15.26:0/2865187444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 msgr2=0x7fcd081100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0d800700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 0x7fcd081100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0d800700 1 -- 172.21.15.26:0/2865187444 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc005040 con 0x7fcd080ff020 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0ce00700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd081002d0 0x7fcd081100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.135+0000 7fcd0d800700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd080ff020 0x7fcd0810fb60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00b3b0 tx=0x7fccfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:36.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.136+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccfc00e040 con 0x7fcd080ff020 2024-04-16T07:20:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.136+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccfc00bcc0 con 0x7fcd080ff020 2024-04-16T07:20:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.136+0000 7fcd10200700 1 -- 172.21.15.26:0/2865187444 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd08115210 con 0x7fcd080ff020 2024-04-16T07:20:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.136+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccfc01c630 con 0x7fcd080ff020 2024-04-16T07:20:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.136+0000 7fcd10200700 1 -- 172.21.15.26:0/2865187444 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd081157d0 con 0x7fcd080ff020 2024-04-16T07:20:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.137+0000 7fcd10200700 1 -- 172.21.15.26:0/2865187444 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0804c710 con 0x7fcd080ff020 2024-04-16T07:20:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.138+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fccfc013980 con 0x7fcd080ff020 2024-04-16T07:20:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.139+0000 7fcd05600700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fccf00692a0 0x7fccf006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.139+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fccfc0880c0 con 0x7fcd080ff020 2024-04-16T07:20:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.139+0000 7fcd0ce00700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fccf00692a0 0x7fccf006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.140+0000 7fcd0ce00700 1 --2- 172.21.15.26:0/2865187444 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fccf00692a0 0x7fccf006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009fd0 tx=0x7fccf400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:36.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.144+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fccfc057d10 con 0x7fcd080ff020 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003410 con 0x55fc3f51baa0 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:36.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:36.265+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:36.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.362+0000 7fcd10200700 1 -- 172.21.15.26:0/2865187444 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fcd08107b10 con 0x7fcd080ff020 2024-04-16T07:20:36.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:36.363+0000 7fcd05600700 1 -- 172.21.15.26:0/2865187444 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7fccfc0578a0 con 0x7fcd080ff020 2024-04-16T07:20:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-04-16T07:20:36.757 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:36 smithi026 ceph-mon[28805]: pgmap v247: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:36 smithi104 ceph-mon[36163]: pgmap v247: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:36.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:36.975+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:37.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:37.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:37.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:37.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0037b0 con 0x55fc3f51baa0 2024-04-16T07:20:37.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:37.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:37.266+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:37.265 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T07:20:37.266 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:20:37.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-16T07:20:37.788 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:37.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:37.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:37.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:37.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:37.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:37.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:37.975+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:38.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:38.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003b50 con 0x55fc3f51baa0 2024-04-16T07:20:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:38.266+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:38.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:38 smithi026 ceph-mon[28805]: pgmap v248: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T07:20:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:38 smithi104 ceph-mon[36163]: pgmap v248: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.975+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.976+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.976+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.976+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:38.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:38.976+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:39.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:39.266+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 -- 172.21.15.26:0/3501750803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0fff70 msgr2=0x7fe90c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 --2- 172.21.15.26:0/3501750803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0fff70 0x7fe90c102390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4004940 tx=0x7fe8f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 -- 172.21.15.26:0/3501750803 shutdown_connections 2024-04-16T07:20:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 --2- 172.21.15.26:0/3501750803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0fff70 0x7fe90c102390 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 --2- 172.21.15.26:0/3501750803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0662f0 0x7fe90c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 -- 172.21.15.26:0/3501750803 >> 172.21.15.26:0/3501750803 conn(0x7fe90c0f74c0 msgr2=0x7fe90c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 -- 172.21.15.26:0/3501750803 shutdown_connections 2024-04-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.824+0000 7fe910a00700 1 -- 172.21.15.26:0/3501750803 wait complete. 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.825+0000 7fe910a00700 1 Processor -- start 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.825+0000 7fe910a00700 1 -- start start 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe910a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe910a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0fff70 0x7fe90c0fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe910a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0fbee0 con 0x7fe90c0662f0 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe910a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0fc050 con 0x7fe90c0fff70 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe909e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe909e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59824/0 (socket says 172.21.15.26:59824) 2024-04-16T07:20:39.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.826+0000 7fe909e00700 1 -- 172.21.15.26:0/1099113019 learned_addr learned my addr 172.21.15.26:0/1099113019 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:39.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe901400700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0fff70 0x7fe90c0fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:39.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe909e00700 1 -- 172.21.15.26:0/1099113019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0fff70 msgr2=0x7fe90c0fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe909e00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0fff70 0x7fe90c0fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe909e00700 1 -- 172.21.15.26:0/1099113019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4005040 con 0x7fe90c0662f0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe909e00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00c890 tx=0x7fe8fc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8fc00a7d0 con 0x7fe90c0662f0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8fc00ae00 con 0x7fe90c0662f0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.827+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0fc330 con 0x7fe90c0662f0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.828+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8fc0113a0 con 0x7fe90c0662f0 2024-04-16T07:20:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.828+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c115aa0 con 0x7fe90c0662f0 2024-04-16T07:20:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.829+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c05f5f0 con 0x7fe90c0662f0 2024-04-16T07:20:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.830+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe8fc012400 con 0x7fe90c0662f0 2024-04-16T07:20:39.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.831+0000 7fe902a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 0x7fe8f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.831+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe8fc0871f0 con 0x7fe90c0662f0 2024-04-16T07:20:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.831+0000 7fe901400700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 0x7fe8f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.832+0000 7fe901400700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 0x7fe8f006b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4004f90 tx=0x7fe8f4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:39.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:39.836+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe8fc053570 con 0x7fe90c0662f0 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:39.976+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:40.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.054+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fe90c1058d0 con 0x7fe90c0662f0 2024-04-16T07:20:40.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.055+0000 7fe902a00700 1 -- 172.21.15.26:0/1099113019 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fe8fc056b90 con 0x7fe90c0662f0 2024-04-16T07:20:40.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 msgr2=0x7fe8f006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:40.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 0x7fe8f006b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4004f90 tx=0x7fe8f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:40.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 msgr2=0x7fe90c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00c890 tx=0x7fe8fc0044c0 comp rx=0 tx=0).stop 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 shutdown_connections 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe8f00692b0 0x7fe8f006b760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe90c0662f0 0x7fe90c0fd460 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.058+0000 7fe910a00700 1 --2- 172.21.15.26:0/1099113019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe90c0fff70 0x7fe90c0fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.059+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 >> 172.21.15.26:0/1099113019 conn(0x7fe90c0f74c0 msgr2=0x7fe90c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.059+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 shutdown_connections 2024-04-16T07:20:40.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:40.059+0000 7fe910a00700 1 -- 172.21.15.26:0/1099113019 wait complete. 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:40.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:40.266+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:40.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:40 smithi026 ceph-mon[28805]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:40 smithi104 ceph-mon[36163]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:40.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:40.976+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:41.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:41.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:41.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:41.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:41.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:41.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:41.267+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:41.443 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:41.976+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:42.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:42.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:42.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:42.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:42.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:42.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:42.267+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:42 smithi026 ceph-mon[28805]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:42.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:42 smithi104 ceph-mon[36163]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:42.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:42.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:42.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:42.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:42.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:42.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:42.976+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:43.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:43.267+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 -- 172.21.15.26:0/1283411356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d80ff020 msgr2=0x7f70d80ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 --2- 172.21.15.26:0/1283411356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d80ff020 0x7f70d80ff430 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f70c8004940 tx=0x7f70c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 -- 172.21.15.26:0/1283411356 shutdown_connections 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 --2- 172.21.15.26:0/1283411356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d81002d0 0x7f70d8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 --2- 172.21.15.26:0/1283411356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d80ff020 0x7f70d80ff430 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 -- 172.21.15.26:0/1283411356 >> 172.21.15.26:0/1283411356 conn(0x7f70d80fa950 msgr2=0x7f70d80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 -- 172.21.15.26:0/1283411356 shutdown_connections 2024-04-16T07:20:43.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.656+0000 7f70de800700 1 -- 172.21.15.26:0/1283411356 wait complete. 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.657+0000 7f70de800700 1 Processor -- start 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.657+0000 7f70de800700 1 -- start start 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70de800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 0x7f70d810fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70de800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70de800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8110720 con 0x7f70d81002d0 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70de800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81150c0 con 0x7f70d80ff020 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59838/0 (socket says 172.21.15.26:59838) 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 -- 172.21.15.26:0/2312309924 learned_addr learned my addr 172.21.15.26:0/2312309924 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d7400700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 0x7f70d810fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 -- 172.21.15.26:0/2312309924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 msgr2=0x7f70d810fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 0x7f70d810fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.658+0000 7f70d6a00700 1 -- 172.21.15.26:0/2312309924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8005040 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70d7400700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 0x7f70d810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70d6a00700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f70c400b790 tx=0x7f70c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c400f770 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70c400fda0 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c400d310 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d8115320 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.659+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d8115890 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.660+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d804c710 con 0x7f70d81002d0 2024-04-16T07:20:43.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.662+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f70c4010420 con 0x7f70d81002d0 2024-04-16T07:20:43.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.663+0000 7f70cf400700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 0x7f70bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:43.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.663+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f70c40871c0 con 0x7f70d81002d0 2024-04-16T07:20:43.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.663+0000 7f70d7400700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 0x7f70bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:43.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.664+0000 7f70d7400700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 0x7f70bc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b7a0 tx=0x7f70c8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:43.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.668+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f70c4056f10 con 0x7f70d81002d0 2024-04-16T07:20:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.889+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f70d8107b90 con 0x7f70d81002d0 2024-04-16T07:20:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.890+0000 7f70cf400700 1 -- 172.21.15.26:0/2312309924 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f70c4056aa0 con 0x7f70d81002d0 2024-04-16T07:20:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 msgr2=0x7f70bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 0x7f70bc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b7a0 tx=0x7f70c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 msgr2=0x7f70d8110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f70c400b790 tx=0x7f70c4009250 comp rx=0 tx=0).stop 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 shutdown_connections 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f70bc0692a0 0x7f70bc06b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70d80ff020 0x7f70d810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 --2- 172.21.15.26:0/2312309924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70d81002d0 0x7f70d8110090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.893+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 >> 172.21.15.26:0/2312309924 conn(0x7f70d80fa950 msgr2=0x7f70d80fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.894+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 shutdown_connections 2024-04-16T07:20:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:43.894+0000 7f70de800700 1 -- 172.21.15.26:0/2312309924 wait complete. 2024-04-16T07:20:43.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.976+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:43.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.976+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.976+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.976+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.976+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 3) v5 -- 0x7fbf68003fa0 con 0x563ccaa1cfa0 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 7 ==== client_session(renewcaps seq 3) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68003fa0 con 0x563ccaa1cfa0 2024-04-16T07:20:43.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:43.977+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 3) v1 from mds.0 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:44.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:44.268+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:44.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:44 smithi026 ceph-mon[28805]: pgmap v251: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:44.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-16T07:20:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:44 smithi104 ceph-mon[36163]: pgmap v251: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:44.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:44.977+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:45.192 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:45.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:45.268+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:45.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:45.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:45.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:45.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:45.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:45.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:45.978+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:46.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:46.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:46.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:46.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:46.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:46.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:46.268+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:46.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:46 smithi026 ceph-mon[28805]: pgmap v252: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:46.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:46 smithi104 ceph-mon[36163]: pgmap v252: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:20:46.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:46.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:46.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:46.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:46.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:46.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:46.977+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:47.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:47.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:47.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:47.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:47.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:47.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:47.268+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 -- 172.21.15.26:0/584015082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740662f0 msgr2=0x7fa2740ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 --2- 172.21.15.26:0/584015082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740662f0 0x7fa2740ffa30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa264004940 tx=0x7fa264009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 -- 172.21.15.26:0/584015082 shutdown_connections 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 --2- 172.21.15.26:0/584015082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740fff70 0x7fa274102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 --2- 172.21.15.26:0/584015082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740662f0 0x7fa2740ffa30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 -- 172.21.15.26:0/584015082 >> 172.21.15.26:0/584015082 conn(0x7fa2740f74c0 msgr2=0x7fa2740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:47.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 -- 172.21.15.26:0/584015082 shutdown_connections 2024-04-16T07:20:47.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.285+0000 7fa279c00700 1 -- 172.21.15.26:0/584015082 wait complete. 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.286+0000 7fa279c00700 1 Processor -- start 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.286+0000 7fa279c00700 1 -- start start 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa279c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740662f0 0x7fa27410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa279c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa279c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274110860 con 0x7fa2740fff70 2024-04-16T07:20:47.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa279c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274110fa0 con 0x7fa2740662f0 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59866/0 (socket says 172.21.15.26:59866) 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 -- 172.21.15.26:0/3538146936 learned_addr learned my addr 172.21.15.26:0/3538146936 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa272a00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740662f0 0x7fa27410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 -- 172.21.15.26:0/3538146936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740662f0 msgr2=0x7fa27410fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740662f0 0x7fa27410fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.287+0000 7fa26a000700 1 -- 172.21.15.26:0/3538146936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264005040 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa26a000700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa25c002a50 tx=0x7fa25c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25c007cc0 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa25c00f040 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25c00da70 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa274111280 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.288+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa274111840 con 0x7fa2740fff70 2024-04-16T07:20:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.289+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27405f5f0 con 0x7fa2740fff70 2024-04-16T07:20:47.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.291+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa25c00dc10 con 0x7fa2740fff70 2024-04-16T07:20:47.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.291+0000 7fa26b400700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 0x7fa25806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:47.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.292+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa25c086880 con 0x7fa2740fff70 2024-04-16T07:20:47.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.292+0000 7fa272a00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 0x7fa25806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:47.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.292+0000 7fa272a00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 0x7fa25806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa26400b970 tx=0x7fa264009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:47.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.296+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa25c056550 con 0x7fa2740fff70 2024-04-16T07:20:47.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.512+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fa2741058e0 con 0x7fa2740fff70 2024-04-16T07:20:47.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.513+0000 7fa26b400700 1 -- 172.21.15.26:0/3538146936 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fa25c0560e0 con 0x7fa2740fff70 2024-04-16T07:20:47.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 msgr2=0x7fa25806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 0x7fa25806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa26400b970 tx=0x7fa264009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 msgr2=0x7fa274110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa25c002a50 tx=0x7fa25c0046b0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 shutdown_connections 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa2580692a0 0x7fa25806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2740662f0 0x7fa27410fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 --2- 172.21.15.26:0/3538146936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2740fff70 0x7fa274110260 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 >> 172.21.15.26:0/3538146936 conn(0x7fa2740f74c0 msgr2=0x7fa2740fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:47.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 shutdown_connections 2024-04-16T07:20:47.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:47.517+0000 7fa279c00700 1 -- 172.21.15.26:0/3538146936 wait complete. 2024-04-16T07:20:47.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:47.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:47.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:47.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:47.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:47.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:47.978+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:48.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:48.268+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:48.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-16T07:20:48.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:48 smithi104 ceph-mon[36163]: pgmap v253: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:48.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:48 smithi026 ceph-mon[28805]: pgmap v253: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T07:20:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:48.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:48.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:49.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:49.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:49.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:49.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:49.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:49.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:49.269+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:49.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:49.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:50.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:50.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:50.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:50.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:50.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:50.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:50.269+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:50 smithi026 ceph-mon[28805]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T07:20:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:50 smithi104 ceph-mon[36163]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T07:20:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 -- 172.21.15.26:0/3849419456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c0fff10 msgr2=0x7fe75c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 --2- 172.21.15.26:0/3849419456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c0fff10 0x7fe75c102330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe748004940 tx=0x7fe748009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 -- 172.21.15.26:0/3849419456 shutdown_connections 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 --2- 172.21.15.26:0/3849419456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c0fff10 0x7fe75c102330 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 --2- 172.21.15.26:0/3849419456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c066b80 0x7fe75c0ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 -- 172.21.15.26:0/3849419456 >> 172.21.15.26:0/3849419456 conn(0x7fe75c0f7440 msgr2=0x7fe75c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.967+0000 7fe763a00700 1 -- 172.21.15.26:0/3849419456 shutdown_connections 2024-04-16T07:20:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.968+0000 7fe763a00700 1 -- 172.21.15.26:0/3849419456 wait complete. 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.968+0000 7fe763a00700 1 Processor -- start 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe763a00700 1 -- start start 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe763a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe763a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c0fff10 0x7fe75c10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe763a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c10e4c0 con 0x7fe75c066b80 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe763a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c10e630 con 0x7fe75c0fff10 2024-04-16T07:20:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe761000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe761000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57920/0 (socket says 172.21.15.26:57920) 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.969+0000 7fe761000700 1 -- 172.21.15.26:0/2202184950 learned_addr learned my addr 172.21.15.26:0/2202184950 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe75be00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c0fff10 0x7fe75c10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe761000700 1 -- 172.21.15.26:0/2202184950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c0fff10 msgr2=0x7fe75c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe761000700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c0fff10 0x7fe75c10dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe761000700 1 -- 172.21.15.26:0/2202184950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748005040 con 0x7fe75c066b80 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe761000700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe750002a50 tx=0x7fe750004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.970+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe750007cc0 con 0x7fe75c066b80 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.971+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c113090 con 0x7fe75c066b80 2024-04-16T07:20:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.971+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe75000f040 con 0x7fe75c066b80 2024-04-16T07:20:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.971+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75000daf0 con 0x7fe75c066b80 2024-04-16T07:20:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.971+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c113650 con 0x7fe75c066b80 2024-04-16T07:20:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.972+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c04c710 con 0x7fe75c066b80 2024-04-16T07:20:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.973+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe750009810 con 0x7fe75c066b80 2024-04-16T07:20:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.974+0000 7fe758c00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 0x7fe74406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.974+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe750087640 con 0x7fe75c066b80 2024-04-16T07:20:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.975+0000 7fe75be00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 0x7fe74406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.975+0000 7fe75be00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 0x7fe74406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe748004f90 tx=0x7fe748009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:50.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe7500573c0 con 0x7fe75c066b80 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:50.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:50.978+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:51.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.212+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}) v1 -- 0x7fe75c0f8f80 con 0x7fe7440692a0 2024-04-16T07:20:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.224+0000 7fe758c00700 1 -- 172.21.15.26:0/2202184950 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fe75c0f8f80 con 0x7fe7440692a0 2024-04-16T07:20:51.223 INFO:teuthology.orchestra.run.smithi026.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:20:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 msgr2=0x7fe74406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 0x7fe74406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe748004f90 tx=0x7fe748009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 msgr2=0x7fe75c10d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe750002a50 tx=0x7fe750004600 comp rx=0 tx=0).stop 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 shutdown_connections 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe7440692a0 0x7fe74406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe75c066b80 0x7fe75c10d980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 --2- 172.21.15.26:0/2202184950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe75c0fff10 0x7fe75c10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 >> 172.21.15.26:0/2202184950 conn(0x7fe75c0f7440 msgr2=0x7fe75c0f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 shutdown_connections 2024-04-16T07:20:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:51.228+0000 7fe763a00700 1 -- 172.21.15.26:0/2202184950 wait complete. 2024-04-16T07:20:51.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:51.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:51.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:51.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:51.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:51.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:51.269+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:51.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:51.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:52.112 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T07:20:52.113 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:20:52.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 3) v5 -- 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:52.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.269+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:52.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.270+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 7 ==== client_session(renewcaps seq 3) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac003ef0 con 0x55fc3f51baa0 2024-04-16T07:20:52.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:52.270+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 3) v1 from mds.0 2024-04-16T07:20:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:52 smithi026 ceph-mon[28805]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:52 smithi026 ceph-mon[28805]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:20:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:52 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:52 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:20:52.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:52 smithi026 ceph-mon[28805]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T07:20:52.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:52 smithi104 ceph-mon[36163]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:52.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:52 smithi104 ceph-mon[36163]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:20:52.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:52 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:52.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:52 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:20:52.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:52 smithi104 ceph-mon[36163]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T07:20:52.625 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:52.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:52.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:53.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:53.269+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:53.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:53.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:53.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:53 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:53.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:53 smithi026 ceph-mon[28805]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:20:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:20:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:20:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:53 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:20:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:53 smithi104 ceph-mon[36163]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:53.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:53.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:54.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:54.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:54.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:54.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:54.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:54.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:54.270+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:54.379 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:54 smithi026 ceph-mon[28805]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:54.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:54 smithi104 ceph-mon[36163]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 -- 172.21.15.26:0/1781979019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 msgr2=0x7f38340fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 --2- 172.21.15.26:0/1781979019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f38340fc150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3824004940 tx=0x7f3824009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 -- 172.21.15.26:0/1781979019 shutdown_connections 2024-04-16T07:20:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 --2- 172.21.15.26:0/1781979019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 0x7f38340fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 --2- 172.21.15.26:0/1781979019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f38340fc150 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:54.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 -- 172.21.15.26:0/1781979019 >> 172.21.15.26:0/1781979019 conn(0x7f38340f74c0 msgr2=0x7f38340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:54.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.731+0000 7f383b000700 1 -- 172.21.15.26:0/1781979019 shutdown_connections 2024-04-16T07:20:54.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.732+0000 7f383b000700 1 -- 172.21.15.26:0/1781979019 wait complete. 2024-04-16T07:20:54.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.732+0000 7f383b000700 1 Processor -- start 2024-04-16T07:20:54.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f383b000700 1 -- start start 2024-04-16T07:20:54.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f383b000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f383b000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 0x7f3834110120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f383b000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3834110720 con 0x7f38340fbd40 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f383b000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3834110f20 con 0x7f38340fcff0 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f3833e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f3833e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57948/0 (socket says 172.21.15.26:57948) 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.733+0000 7f3833e00700 1 -- 172.21.15.26:0/1332161599 learned_addr learned my addr 172.21.15.26:0/1332161599 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.734+0000 7f3833400700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 0x7f3834110120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.734+0000 7f3833e00700 1 -- 172.21.15.26:0/1332161599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 msgr2=0x7f3834110120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.734+0000 7f3833e00700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 0x7f3834110120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.734+0000 7f3833e00700 1 -- 172.21.15.26:0/1332161599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824005040 con 0x7f38340fbd40 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.734+0000 7f3833e00700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f382400b0d0 tx=0x7f3824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:54.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.735+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f382400e030 con 0x7f38340fbd40 2024-04-16T07:20:54.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.735+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f382401c400 con 0x7f38340fbd40 2024-04-16T07:20:54.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.735+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f382400b650 con 0x7f38340fbd40 2024-04-16T07:20:54.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.735+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38341111a0 con 0x7f38340fbd40 2024-04-16T07:20:54.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.735+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38341116e0 con 0x7f38340fbd40 2024-04-16T07:20:54.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.736+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834101980 con 0x7f38340fbd40 2024-04-16T07:20:54.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.737+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f382401c570 con 0x7f38340fbd40 2024-04-16T07:20:54.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.738+0000 7f382be00700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:54.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.738+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3824088a00 con 0x7f38340fbd40 2024-04-16T07:20:54.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.739+0000 7f3833400700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:54.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.739+0000 7f3833400700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 0x7f381806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3820004600 tx=0x7f382000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:54.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.743+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3824058750 con 0x7f38340fbd40 2024-04-16T07:20:54.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:54.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:54.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:54.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:54.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:54.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:54.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:54.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:54.996+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38341119c0 con 0x7f38180692a0 2024-04-16T07:20:54.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.001+0000 7f382be00700 1 -- 172.21.15.26:0/1332161599 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f38341119c0 con 0x7f38180692a0 2024-04-16T07:20:55.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.003+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 msgr2=0x7f381806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.003+0000 7f383b000700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 0x7f381806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3820004600 tx=0x7f382000b040 comp rx=0 tx=0).stop 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.003+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 msgr2=0x7f383410fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.003+0000 7f383b000700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f382400b0d0 tx=0x7f3824009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 shutdown_connections 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38340fbd40 0x7f383410fbe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 --2- 172.21.15.26:0/1332161599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38340fcff0 0x7f3834110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 >> 172.21.15.26:0/1332161599 conn(0x7f38340f74c0 msgr2=0x7f3834065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 shutdown_connections 2024-04-16T07:20:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.004+0000 7f383b000700 1 -- 172.21.15.26:0/1332161599 wait complete. 2024-04-16T07:20:55.019 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:20:55.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.169+0000 7f97d6000700 1 -- 172.21.15.26:0/3742918279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 msgr2=0x7f97d0103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.169+0000 7f97d6000700 1 --2- 172.21.15.26:0/3742918279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0103680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f97c4004940 tx=0x7f97c4009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 -- 172.21.15.26:0/3742918279 shutdown_connections 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 --2- 172.21.15.26:0/3742918279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 0x7f97d0104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 --2- 172.21.15.26:0/3742918279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0103680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 -- 172.21.15.26:0/3742918279 >> 172.21.15.26:0/3742918279 conn(0x7f97d0072530 msgr2=0x7f97d0074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 -- 172.21.15.26:0/3742918279 shutdown_connections 2024-04-16T07:20:55.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.170+0000 7f97d6000700 1 -- 172.21.15.26:0/3742918279 wait complete. 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.171+0000 7f97d6000700 1 Processor -- start 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d6000700 1 -- start start 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d6000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 0x7f97d0114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d6000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d0114a60 con 0x7f97d0103270 2024-04-16T07:20:55.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d6000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d01151a0 con 0x7f97d0104520 2024-04-16T07:20:55.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d4c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d4c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57968/0 (socket says 172.21.15.26:57968) 2024-04-16T07:20:55.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.172+0000 7f97d4c00700 1 -- 172.21.15.26:0/1904084212 learned_addr learned my addr 172.21.15.26:0/1904084212 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:55.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.173+0000 7f97cfe00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 0x7f97d0114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.173+0000 7f97d4c00700 1 -- 172.21.15.26:0/1904084212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 msgr2=0x7f97d0114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.173+0000 7f97d4c00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 0x7f97d0114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.173+0000 7f97d4c00700 1 -- 172.21.15.26:0/1904084212 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c4005040 con 0x7f97d0103270 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.173+0000 7f97d4c00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b3b0 tx=0x7f97c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.174+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c400e040 con 0x7f97d0103270 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.174+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97c4004290 con 0x7f97d0103270 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.174+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c401c630 con 0x7f97d0103270 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.174+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d0115420 con 0x7f97d0103270 2024-04-16T07:20:55.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.174+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d01159e0 con 0x7f97d0103270 2024-04-16T07:20:55.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.176+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d00760a0 con 0x7f97d0103270 2024-04-16T07:20:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.176+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f97c400bbf0 con 0x7f97d0103270 2024-04-16T07:20:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.177+0000 7f97ccc00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 0x7f97b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.178+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f97c4088660 con 0x7f97d0103270 2024-04-16T07:20:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.178+0000 7f97cfe00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 0x7f97b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.178+0000 7f97cfe00700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 0x7f97b806b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fd0 tx=0x7f97bc009f20 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.182+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f97c40583b0 con 0x7f97d0103270 2024-04-16T07:20:55.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:55.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:55.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:55.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:55.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:55.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:55.270+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:55.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.444+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97d0115d60 con 0x7f97b80692a0 2024-04-16T07:20:55.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.445+0000 7f97ccc00700 1 -- 172.21.15.26:0/1904084212 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f97d0115d60 con 0x7f97b80692a0 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.448+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 msgr2=0x7f97b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.448+0000 7f97d6000700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 0x7f97b806b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fd0 tx=0x7f97bc009f20 comp rx=0 tx=0).stop 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 msgr2=0x7f97d0113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b3b0 tx=0x7f97c4009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 shutdown_connections 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f97b80692a0 0x7f97b806b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97d0103270 0x7f97d0113f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 --2- 172.21.15.26:0/1904084212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d0104520 0x7f97d0114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 >> 172.21.15.26:0/1904084212 conn(0x7f97d0072530 msgr2=0x7f97d0074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 shutdown_connections 2024-04-16T07:20:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.449+0000 7f97d6000700 1 -- 172.21.15.26:0/1904084212 wait complete. 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.613+0000 7ff235800700 1 -- 172.21.15.26:0/1302803280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300fff70 msgr2=0x7ff230102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.613+0000 7ff235800700 1 --2- 172.21.15.26:0/1302803280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300fff70 0x7ff230102390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004940 tx=0x7ff21c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 -- 172.21.15.26:0/1302803280 shutdown_connections 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 --2- 172.21.15.26:0/1302803280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300fff70 0x7ff230102390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 --2- 172.21.15.26:0/1302803280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300662f0 0x7ff2300ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 -- 172.21.15.26:0/1302803280 >> 172.21.15.26:0/1302803280 conn(0x7ff2300f74c0 msgr2=0x7ff2300f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 -- 172.21.15.26:0/1302803280 shutdown_connections 2024-04-16T07:20:55.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.614+0000 7ff235800700 1 -- 172.21.15.26:0/1302803280 wait complete. 2024-04-16T07:20:55.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.615+0000 7ff235800700 1 Processor -- start 2024-04-16T07:20:55.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.615+0000 7ff235800700 1 -- start start 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff235800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff235800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300fff70 0x7ff2300fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff235800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300fbee0 con 0x7ff2300662f0 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff235800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300fc050 con 0x7ff2300fff70 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57992/0 (socket says 172.21.15.26:57992) 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 -- 172.21.15.26:0/1304626724 learned_addr learned my addr 172.21.15.26:0/1304626724 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22e000700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300fff70 0x7ff2300fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 -- 172.21.15.26:0/1304626724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300fff70 msgr2=0x7ff2300fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300fff70 0x7ff2300fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.616+0000 7ff22ea00700 1 -- 172.21.15.26:0/1304626724 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c005040 con 0x7ff2300662f0 2024-04-16T07:20:55.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff22ea00700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff220002a80 tx=0x7ff220004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:55.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff220007cc0 con 0x7ff2300662f0 2024-04-16T07:20:55.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff22000f040 con 0x7ff2300662f0 2024-04-16T07:20:55.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22000daf0 con 0x7ff2300662f0 2024-04-16T07:20:55.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2300fc330 con 0x7ff2300662f0 2024-04-16T07:20:55.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.617+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff230115a80 con 0x7ff2300662f0 2024-04-16T07:20:55.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.619+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23005f5f0 con 0x7ff2300662f0 2024-04-16T07:20:55.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.621+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff22000dc90 con 0x7ff2300662f0 2024-04-16T07:20:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.622+0000 7ff226a00700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 0x7ff21406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.622+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff2200869e0 con 0x7ff2300662f0 2024-04-16T07:20:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.622+0000 7ff22e000700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 0x7ff21406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:55.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.623+0000 7ff22e000700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 0x7ff21406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004f90 tx=0x7ff21c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:55.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.628+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff22004d250 con 0x7ff2300662f0 2024-04-16T07:20:55.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.890+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2300f8fb0 con 0x7ff2140692b0 2024-04-16T07:20:55.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.915+0000 7ff226a00700 1 -- 172.21.15.26:0/1304626724 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7ff2300f8fb0 con 0x7ff2140692b0 2024-04-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (6m) 41s ago 8m 13.1M - 0.25.0 c8568f914cd2 d5f7c216b68d 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (8m) 41s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (7m) 47s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (6m) 41s ago 7m 73.0M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (73s) 41s ago 72s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (64s) 41s ago 63s 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (69s) 47s ago 68s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (60s) 47s ago 59s 10.4M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (9m) 41s ago 9m 481M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (6m) 47s ago 6m 432M - 17.2.7-847-g45317f95 b333581ccdf5 275c78dfc5a9 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (10m) 41s ago 10m 44.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (6m) 47s ago 6m 31.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (8m) 41s ago 8m 8074k - 1.5.0 0da6a335fe13 5198001b496e 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 47s ago 7m 7503k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (5m) 41s ago 5m 43.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (5m) 41s ago 5m 46.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (4m) 41s ago 4m 44.1M 4290M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (4m) 47s ago 4m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (3m) 47s ago 3m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (2m) 47s ago 2m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (6m) 41s ago 7m 46.4M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 msgr2=0x7ff21406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 0x7ff21406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004f90 tx=0x7ff21c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 msgr2=0x7ff2300fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff220002a80 tx=0x7ff220004630 comp rx=0 tx=0).stop 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 shutdown_connections 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7ff2140692b0 0x7ff21406b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2300662f0 0x7ff2300fd460 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 --2- 172.21.15.26:0/1304626724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2300fff70 0x7ff2300fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 >> 172.21.15.26:0/1304626724 conn(0x7ff2300f74c0 msgr2=0x7ff2300f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 shutdown_connections 2024-04-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:55.919+0000 7ff235800700 1 -- 172.21.15.26:0/1304626724 wait complete. 2024-04-16T07:20:55.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:55.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:55.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:55.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:55.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:55.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:55.979+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 -- 172.21.15.26:0/1942294118 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 msgr2=0x7f03380ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 --2- 172.21.15.26:0/1942294118 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f03380ff410 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0328004940 tx=0x7f0328009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 -- 172.21.15.26:0/1942294118 shutdown_connections 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 --2- 172.21.15.26:0/1942294118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 0x7f0338100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 --2- 172.21.15.26:0/1942294118 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f03380ff410 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.092+0000 7f033de00700 1 -- 172.21.15.26:0/1942294118 >> 172.21.15.26:0/1942294118 conn(0x7f03380fa950 msgr2=0x7f03380fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.093+0000 7f033de00700 1 -- 172.21.15.26:0/1942294118 shutdown_connections 2024-04-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.093+0000 7f033de00700 1 -- 172.21.15.26:0/1942294118 wait complete. 2024-04-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.094+0000 7f033de00700 1 Processor -- start 2024-04-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.094+0000 7f033de00700 1 -- start start 2024-04-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f033de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f033de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 0x7f0338110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f033de00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338110770 con 0x7f03380ff000 2024-04-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f033de00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338110fc0 con 0x7f03381002b0 2024-04-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f0336a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f0336a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58018/0 (socket says 172.21.15.26:58018) 2024-04-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f0336a00700 1 -- 172.21.15.26:0/682249644 learned_addr learned my addr 172.21.15.26:0/682249644 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.095+0000 7f0336000700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 0x7f0338110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f0336a00700 1 -- 172.21.15.26:0/682249644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 msgr2=0x7f0338110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f0336a00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 0x7f0338110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f0336a00700 1 -- 172.21.15.26:0/682249644 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328005040 con 0x7f03380ff000 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f0336a00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f032800b3b0 tx=0x7f0328009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f032800e040 con 0x7f03380ff000 2024-04-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.096+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0328004290 con 0x7f03380ff000 2024-04-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.097+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0338111240 con 0x7f03380ff000 2024-04-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.097+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f032801c630 con 0x7f03380ff000 2024-04-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.097+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0338111780 con 0x7f03380ff000 2024-04-16T07:20:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.098+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033805f5f0 con 0x7f03380ff000 2024-04-16T07:20:56.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.100+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f032800bbf0 con 0x7f03380ff000 2024-04-16T07:20:56.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.101+0000 7f032ea00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 0x7f031c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.101+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f03280890c0 con 0x7f03380ff000 2024-04-16T07:20:56.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.102+0000 7f0336000700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 0x7f031c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.103+0000 7f0336000700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 0x7f031c06b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f0324009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:56.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.106+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0328058ed0 con 0x7f03380ff000 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:56.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:56.271+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:56.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:56 smithi026 ceph-mon[28805]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:56.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:56 smithi026 ceph-mon[28805]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:56.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:56 smithi026 ceph-mon[28805]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:56.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.453+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0338107af0 con 0x7f03380ff000 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.454+0000 7f032ea00700 1 -- 172.21.15.26:0/682249644 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f0328058a60 con 0x7f03380ff000 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:56.453 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:20:56.454 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:20:56.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.457+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 msgr2=0x7f031c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.457+0000 7f033de00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 0x7f031c06b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f0324009040 comp rx=0 tx=0).stop 2024-04-16T07:20:56.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 msgr2=0x7f033810fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f032800b3b0 tx=0x7f0328009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 shutdown_connections 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f031c0692b0 0x7f031c06b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03380ff000 0x7f033810fc30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 --2- 172.21.15.26:0/682249644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03381002b0 0x7f0338110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 >> 172.21.15.26:0/682249644 conn(0x7f03380fa950 msgr2=0x7f03380fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 shutdown_connections 2024-04-16T07:20:56.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.458+0000 7f033de00700 1 -- 172.21.15.26:0/682249644 wait complete. 2024-04-16T07:20:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 -- 172.21.15.26:0/493594667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380ffd40 msgr2=0x7fe538102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 --2- 172.21.15.26:0/493594667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380ffd40 0x7fe538102160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe524004940 tx=0x7fe524009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 -- 172.21.15.26:0/493594667 shutdown_connections 2024-04-16T07:20:56.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 --2- 172.21.15.26:0/493594667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380ffd40 0x7fe538102160 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 --2- 172.21.15.26:0/493594667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380662f0 0x7fe5380ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 -- 172.21.15.26:0/493594667 >> 172.21.15.26:0/493594667 conn(0x7fe5380f7270 msgr2=0x7fe5380f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:56.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 -- 172.21.15.26:0/493594667 shutdown_connections 2024-04-16T07:20:56.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.616+0000 7fe53e600700 1 -- 172.21.15.26:0/493594667 wait complete. 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.617+0000 7fe53e600700 1 Processor -- start 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.617+0000 7fe53e600700 1 -- start start 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53e600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53e600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380ffd40 0x7fe53810dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53e600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53810e3e0 con 0x7fe5380662f0 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53e600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53810ec30 con 0x7fe5380ffd40 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53d200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53d200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58038/0 (socket says 172.21.15.26:58038) 2024-04-16T07:20:56.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.618+0000 7fe53d200700 1 -- 172.21.15.26:0/3889731194 learned_addr learned my addr 172.21.15.26:0/3889731194 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:56.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe53d200700 1 -- 172.21.15.26:0/3889731194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380ffd40 msgr2=0x7fe53810dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe537e00700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380ffd40 0x7fe53810dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe53d200700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380ffd40 0x7fe53810dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe53d200700 1 -- 172.21.15.26:0/3889731194 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524005040 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe53d200700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe52c002a50 tx=0x7fe52c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52c007cc0 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe52c00f040 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.619+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52c00daf0 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.620+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53810ef10 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.620+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53810f450 con 0x7fe5380662f0 2024-04-16T07:20:56.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.621+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53805f5f0 con 0x7fe5380662f0 2024-04-16T07:20:56.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.624+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe52c009810 con 0x7fe5380662f0 2024-04-16T07:20:56.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.624+0000 7fe534c00700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 0x7fe52006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:56.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.624+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe52c086ed0 con 0x7fe5380662f0 2024-04-16T07:20:56.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.625+0000 7fe537e00700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 0x7fe52006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:56.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.626+0000 7fe537e00700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 0x7fe52006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b970 tx=0x7fe524009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:56.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.634+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe52c056c40 con 0x7fe5380662f0 2024-04-16T07:20:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:56 smithi104 ceph-mon[36163]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:56 smithi104 ceph-mon[36163]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:56 smithi104 ceph-mon[36163]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.918+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe53810f0a0 con 0x7fe5380662f0 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.919+0000 7fe534c00700 1 -- 172.21.15.26:0/3889731194 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7fe53810f0a0 con 0x7fe5380662f0 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:20:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:20:56.919 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:56.921 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 msgr2=0x7fe52006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 0x7fe52006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b970 tx=0x7fe524009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 msgr2=0x7fe53810d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe52c002a50 tx=0x7fe52c004600 comp rx=0 tx=0).stop 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 shutdown_connections 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fe5200692b0 0x7fe52006b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5380662f0 0x7fe53810d8a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 --2- 172.21.15.26:0/3889731194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5380ffd40 0x7fe53810dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 >> 172.21.15.26:0/3889731194 conn(0x7fe5380f7270 msgr2=0x7fe5380f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 shutdown_connections 2024-04-16T07:20:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:56.922+0000 7fe53e600700 1 -- 172.21.15.26:0/3889731194 wait complete. 2024-04-16T07:20:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:20:56.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.979+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:56.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.979+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:56.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.979+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:56.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.979+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:56.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.980+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:56.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:56.980+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 -- 172.21.15.26:0/3415121280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700fff70 msgr2=0x7f3c70102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 --2- 172.21.15.26:0/3415121280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700fff70 0x7f3c70102390 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3c58004940 tx=0x7f3c58009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 -- 172.21.15.26:0/3415121280 shutdown_connections 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 --2- 172.21.15.26:0/3415121280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700fff70 0x7f3c70102390 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 --2- 172.21.15.26:0/3415121280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700662f0 0x7f3c700ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 -- 172.21.15.26:0/3415121280 >> 172.21.15.26:0/3415121280 conn(0x7f3c700f74c0 msgr2=0x7f3c700f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 -- 172.21.15.26:0/3415121280 shutdown_connections 2024-04-16T07:20:57.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.079+0000 7f3c76800700 1 -- 172.21.15.26:0/3415121280 wait complete. 2024-04-16T07:20:57.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.080+0000 7f3c76800700 1 Processor -- start 2024-04-16T07:20:57.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.080+0000 7f3c76800700 1 -- start start 2024-04-16T07:20:57.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c76800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c76800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700fff70 0x7f3c70110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c76800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70110850 con 0x7f3c700662f0 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c76800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70110f90 con 0x7f3c700fff70 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c6f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c6f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58064/0 (socket says 172.21.15.26:58064) 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c6f400700 1 -- 172.21.15.26:0/815838461 learned_addr learned my addr 172.21.15.26:0/815838461 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.081+0000 7f3c66a00700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700fff70 0x7f3c70110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c6f400700 1 -- 172.21.15.26:0/815838461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700fff70 msgr2=0x7f3c70110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c6f400700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700fff70 0x7f3c70110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c6f400700 1 -- 172.21.15.26:0/815838461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58005040 con 0x7f3c700662f0 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c6f400700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c60001d00 tx=0x7f3c60009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c60007d10 con 0x7f3c700662f0 2024-04-16T07:20:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c60007e70 con 0x7f3c700662f0 2024-04-16T07:20:57.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c60014400 con 0x7f3c700662f0 2024-04-16T07:20:57.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.082+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c70111270 con 0x7f3c700662f0 2024-04-16T07:20:57.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.083+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c70111830 con 0x7f3c700662f0 2024-04-16T07:20:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.084+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7005f5f0 con 0x7f3c700662f0 2024-04-16T07:20:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.085+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f3c600091d0 con 0x7f3c700662f0 2024-04-16T07:20:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.086+0000 7f3c67e00700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 0x7f3c5406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.086+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3c60087080 con 0x7f3c700662f0 2024-04-16T07:20:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.087+0000 7f3c66a00700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 0x7f3c5406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.088+0000 7f3c66a00700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 0x7f3c5406b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800b3b0 tx=0x7f3c58009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:57.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.091+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3c60056e00 con 0x7f3c700662f0 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:57.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:57.270+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:57.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.349+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c70111b60 con 0x7f3c540692b0 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.350+0000 7f3c67e00700 1 -- 172.21.15.26:0/815838461 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f3c70111b60 con 0x7f3c540692b0 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "", 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 image", 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:20:57.349 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 msgr2=0x7f3c5406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 0x7f3c5406b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800b3b0 tx=0x7f3c58009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 msgr2=0x7f3c7010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c60001d00 tx=0x7f3c60009e40 comp rx=0 tx=0).stop 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 shutdown_connections 2024-04-16T07:20:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f3c540692b0 0x7f3c5406b760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c700662f0 0x7f3c7010fd10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 --2- 172.21.15.26:0/815838461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c700fff70 0x7f3c70110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.354+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 >> 172.21.15.26:0/815838461 conn(0x7f3c700f74c0 msgr2=0x7f3c700fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.355+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 shutdown_connections 2024-04-16T07:20:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.355+0000 7f3c76800700 1 -- 172.21.15.26:0/815838461 wait complete. 2024-04-16T07:20:57.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 -- 172.21.15.26:0/2431170112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 msgr2=0x7f31800fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 --2- 172.21.15.26:0/2431170112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f31800fe070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f316c004940 tx=0x7f316c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 -- 172.21.15.26:0/2431170112 shutdown_connections 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 --2- 172.21.15.26:0/2431170112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f31800fe070 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 --2- 172.21.15.26:0/2431170112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 0x7f31800fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 -- 172.21.15.26:0/2431170112 >> 172.21.15.26:0/2431170112 conn(0x7f31800f8540 msgr2=0x7f31800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 -- 172.21.15.26:0/2431170112 shutdown_connections 2024-04-16T07:20:57.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.510+0000 7f3184c00700 1 -- 172.21.15.26:0/2431170112 wait complete. 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.511+0000 7f3184c00700 1 Processor -- start 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.511+0000 7f3184c00700 1 -- start start 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.511+0000 7f3184c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 0x7f318010dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f3184c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f3184c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010e5f0 con 0x7f31800fd710 2024-04-16T07:20:57.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f3184c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010ee40 con 0x7f31800fcdc0 2024-04-16T07:20:57.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f317d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f317d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58088/0 (socket says 172.21.15.26:58088) 2024-04-16T07:20:57.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f317d400700 1 -- 172.21.15.26:0/3171911675 learned_addr learned my addr 172.21.15.26:0/3171911675 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:20:57.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.512+0000 7f317de00700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 0x7f318010dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f317d400700 1 -- 172.21.15.26:0/3171911675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 msgr2=0x7f318010dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f317d400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 0x7f318010dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f317d400700 1 -- 172.21.15.26:0/3171911675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c005040 con 0x7f31800fd710 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f317d400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f316c00b970 tx=0x7f316c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316c00e040 con 0x7f31800fd710 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f3184c00700 1 -- 172.21.15.26:0/3171911675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318010f0c0 con 0x7f31800fd710 2024-04-16T07:20:57.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f316c004290 con 0x7f31800fd710 2024-04-16T07:20:57.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.513+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316c01c3f0 con 0x7f31800fd710 2024-04-16T07:20:57.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.514+0000 7f3184c00700 1 -- 172.21.15.26:0/3171911675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318010f600 con 0x7f31800fd710 2024-04-16T07:20:57.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.515+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318005f5f0 con 0x7f31800fd710 2024-04-16T07:20:57.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.517+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f316c00bbf0 con 0x7f31800fd710 2024-04-16T07:20:57.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.519+0000 7f3176000700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 0x7f316406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:20:57.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.519+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f316c088a10 con 0x7f31800fd710 2024-04-16T07:20:57.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.519+0000 7f317de00700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 0x7f316406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:20:57.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.520+0000 7f317de00700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 0x7f316406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31800fdbc0 tx=0x7f317000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:20:57.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.522+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f316c058760 con 0x7f31800fd710 2024-04-16T07:20:57.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:57 smithi104 ceph-mon[36163]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:57.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:57 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/682249644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:20:57.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:57 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3889731194' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:20:57.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:57 smithi026 ceph-mon[28805]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:57.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:57 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/682249644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:20:57.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:57 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3889731194' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:20:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.846+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f318004bed0 con 0x7f31800fd710 2024-04-16T07:20:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.847+0000 7f3176000700 1 -- 172.21.15.26:0/3171911675 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f316c0582f0 con 0x7f31800fd710 2024-04-16T07:20:57.846 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:20:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 msgr2=0x7f316406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 0x7f316406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31800fdbc0 tx=0x7f317000b040 comp rx=0 tx=0).stop 2024-04-16T07:20:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 msgr2=0x7f318010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:20:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f316c00b970 tx=0x7f316c009e30 comp rx=0 tx=0).stop 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 shutdown_connections 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f31640692b0 0x7f316406b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31800fcdc0 0x7f318010dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 --2- 172.21.15.26:0/3171911675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800fd710 0x7f318010dff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 >> 172.21.15.26:0/3171911675 conn(0x7f31800f8540 msgr2=0x7f31800f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 shutdown_connections 2024-04-16T07:20:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:20:57.850+0000 7f316b400700 1 -- 172.21.15.26:0/3171911675 wait complete. 2024-04-16T07:20:57.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:57.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:57.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:57.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:57.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:57.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:57.980+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:58.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:58.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:58.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:58.271+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:58 smithi026 ceph-mon[28805]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:58.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:58 smithi026 ceph-mon[28805]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:58.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:20:58 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3171911675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:20:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:58 smithi104 ceph-mon[36163]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:20:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:58 smithi104 ceph-mon[36163]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:20:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:20:58 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3171911675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:58.980+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:59.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:20:59.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:20:59.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:20:59.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:20:59.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:20:59.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:20:59.271+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:20:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:20:59.980+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:00.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:00.271+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:00.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:00 smithi026 ceph-mon[28805]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:00 smithi104 ceph-mon[36163]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:00.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:00.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:00.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:00.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:00.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:00.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:00.980+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:01.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:01.271+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:01.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:01.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:01.981+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:02.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:02.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:02.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:02.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:02.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:02.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:02.272+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:02 smithi026 ceph-mon[28805]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:02 smithi104 ceph-mon[36163]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:02.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:02.981+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:03.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:03.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:03.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:03.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:03.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:03.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:03.272+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 4) v5 -- 0x7fbf680041e0 con 0x563ccaa1cfa0 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:03.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:03.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.981+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:03.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.982+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 8 ==== client_session(renewcaps seq 4) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680041e0 con 0x563ccaa1cfa0 2024-04-16T07:21:03.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:03.982+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 4) v1 from mds.0 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:04.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:04.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:04.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:04 smithi026 ceph-mon[28805]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:04 smithi104 ceph-mon[36163]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:04.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:04.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:04.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:04.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:04.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:04.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:04.981+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:05.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:05.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:05.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:05.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:05.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:05.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:05.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:05.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.981+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:05.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.981+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:05.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.981+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:05.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.981+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:05.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.982+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:05.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:05.982+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:06.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:06.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:06.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:06.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:06.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:06.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:06.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:06.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:06 smithi026 ceph-mon[28805]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:06.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:06 smithi104 ceph-mon[36163]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x563ccaa1cfa0 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:06.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:06.982+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:07.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:07.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:07.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:07.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:07.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:07.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:07.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:07.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:07.982+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:08.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:08.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:08.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:08 smithi026 ceph-mon[28805]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:08 smithi104 ceph-mon[36163]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:08.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:08.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:08.982+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:09.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:09.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:09.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:09.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:09.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:09.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:09.273+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:09.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.982+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:09.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.982+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:09.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.982+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:09.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.982+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:09.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.983+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:09.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:09.983+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:10.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:10.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:10.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:10.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:10.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:10.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:10.274+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:10 smithi026 ceph-mon[28805]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:10 smithi104 ceph-mon[36163]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:10.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:10.983+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:11.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:11.275+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:11.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:11.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:11.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:11.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:11.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:11.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:11.983+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:12.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:11 smithi026 ceph-mon[28805]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:11 smithi104 ceph-mon[36163]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 4) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:12.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:12.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:12.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:12.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:12.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 8 ==== client_session(renewcaps seq 4) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac001d80 con 0x55fc3f51baa0 2024-04-16T07:21:12.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:12.275+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 4) v1 from mds.0 2024-04-16T07:21:12.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:12.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:12.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:12.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:12.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:12.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:12.983+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:13.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:13.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:13.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:13.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:13.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:13.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:13.275+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:13.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:13.984+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:14.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:14.275+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:14 smithi026 ceph-mon[28805]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:14 smithi104 ceph-mon[36163]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:14.984+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:15.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:15.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:15.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:15.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:15.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:15.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:15.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:15.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:15.984+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:16.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:16.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:16 smithi026 ceph-mon[28805]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:16 smithi104 ceph-mon[36163]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:16.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:16.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:16.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:16.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:16.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:16.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:16.984+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:17.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.274+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:17.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.274+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:17.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.274+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:17.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.274+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:17.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.275+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:17.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:17.275+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:17.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:17.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:17.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:17.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:17.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:17.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:17.985+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:18.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:18.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:18 smithi104 ceph-mon[36163]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:18 smithi026 ceph-mon[28805]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:18.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:18.985+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:19.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:19.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:19.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:19.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:19.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:19.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:19.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:19.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:19.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:19.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:19.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:19.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:19.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:19.985+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:20.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:20.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:20.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:20.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:20.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:20.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:20.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:20 smithi026 ceph-mon[28805]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:20 smithi104 ceph-mon[36163]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:20.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:20.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:20.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:20.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:20.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:20.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:20.985+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:21.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:21.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:21.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:21.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:21.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:21.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:21.277+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:21.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:21.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:21.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:21.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:21.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:21.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:21.986+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:22.277+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:22 smithi026 ceph-mon[28805]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:22 smithi104 ceph-mon[36163]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:22.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:22.986+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:23.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:23.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:23.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:23.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:23.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:23.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:23.276+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 5) v5 -- 0x7fbf680043a0 con 0x563ccaa1cfa0 2024-04-16T07:21:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.986+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.987+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 9 ==== client_session(renewcaps seq 5) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680043a0 con 0x563ccaa1cfa0 2024-04-16T07:21:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:23.987+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 5) v1 from mds.0 2024-04-16T07:21:24.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:24.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:24.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:24.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:24.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:24.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:24.277+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:24.563 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:24 smithi026 ceph-mon[28805]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:24 smithi104 ceph-mon[36163]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:24.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:24.986+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:25.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:25.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:25.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:25.278+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:25.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.986+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:25.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.986+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:25.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.986+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:25.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.986+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:25.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.987+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:25.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:25.987+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:26.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:26.278+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:26 smithi026 ceph-mon[28805]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:26 smithi104 ceph-mon[36163]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:26.987+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:27.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:27.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:27.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:27.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:27.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:27.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:27.278+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:27.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:27.987+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 -- 172.21.15.26:0/1316524535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 msgr2=0x7f0104070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 --2- 172.21.15.26:0/1316524535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f0104070d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00f0004940 tx=0x7f00f0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 -- 172.21.15.26:0/1316524535 shutdown_connections 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 --2- 172.21.15.26:0/1316524535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f0104070d60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 --2- 172.21.15.26:0/1316524535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 0x7f01040703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 -- 172.21.15.26:0/1316524535 >> 172.21.15.26:0/1316524535 conn(0x7f01040f83a0 msgr2=0x7f01040fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:28.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 -- 172.21.15.26:0/1316524535 shutdown_connections 2024-04-16T07:21:28.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.038+0000 7f0103e00700 1 -- 172.21.15.26:0/1316524535 wait complete. 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.039+0000 7f0103e00700 1 Processor -- start 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.039+0000 7f0103e00700 1 -- start start 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0103e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0103e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 0x7f010410ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0103e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104110580 con 0x7f01040708f0 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0103e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104110dd0 con 0x7f0104071ea0 2024-04-16T07:21:28.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0102a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0102a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51146/0 (socket says 172.21.15.26:51146) 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.040+0000 7f0102a00700 1 -- 172.21.15.26:0/1483344221 learned_addr learned my addr 172.21.15.26:0/1483344221 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.041+0000 7f0102000700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 0x7f010410ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.041+0000 7f0102a00700 1 -- 172.21.15.26:0/1483344221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 msgr2=0x7f010410ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.041+0000 7f0102a00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 0x7f010410ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.041+0000 7f0102a00700 1 -- 172.21.15.26:0/1483344221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0005040 con 0x7f01040708f0 2024-04-16T07:21:28.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.041+0000 7f0102a00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00f4002a80 tx=0x7f00f4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:28.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.042+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f4007cc0 con 0x7f01040708f0 2024-04-16T07:21:28.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.042+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00f400f040 con 0x7f01040708f0 2024-04-16T07:21:28.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.042+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01041110b0 con 0x7f01040708f0 2024-04-16T07:21:28.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.042+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01041115f0 con 0x7f01040708f0 2024-04-16T07:21:28.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.042+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f400dad0 con 0x7f01040708f0 2024-04-16T07:21:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.043+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e40052e0 con 0x7f01040708f0 2024-04-16T07:21:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.044+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f00f400dc30 con 0x7f01040708f0 2024-04-16T07:21:28.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.045+0000 7f00faa00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 0x7f00e806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.045+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f00f4086eb0 con 0x7f01040708f0 2024-04-16T07:21:28.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.046+0000 7f0102000700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 0x7f00e806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:28.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.046+0000 7f0102000700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 0x7f00e806b810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f00f0004f90 tx=0x7f00f0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.051+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f00f4056c00 con 0x7f01040708f0 2024-04-16T07:21:28.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:28.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:28.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:28.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:28.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:28.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:28.278+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:28.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.389+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00e4000bf0 con 0x7f00e8069360 2024-04-16T07:21:28.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.393+0000 7f00faa00700 1 -- 172.21.15.26:0/1483344221 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f00e4000bf0 con 0x7f00e8069360 2024-04-16T07:21:28.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 msgr2=0x7f00e806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 0x7f00e806b810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f00f0004f90 tx=0x7f00f0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 msgr2=0x7f010410fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00f4002a80 tx=0x7f00f4004600 comp rx=0 tx=0).stop 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 shutdown_connections 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f00e8069360 0x7f00e806b810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01040708f0 0x7f010410fa40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 --2- 172.21.15.26:0/1483344221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0104071ea0 0x7f010410ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.396+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 >> 172.21.15.26:0/1483344221 conn(0x7f01040f83a0 msgr2=0x7f01040f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.397+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 shutdown_connections 2024-04-16T07:21:28.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.397+0000 7f0103e00700 1 -- 172.21.15.26:0/1483344221 wait complete. 2024-04-16T07:21:28.417 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 -- 172.21.15.26:0/1947128683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc071e40 msgr2=0x7f55fc070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 --2- 172.21.15.26:0/1947128683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc071e40 0x7f55fc070350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f55ec004940 tx=0x7f55ec009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 -- 172.21.15.26:0/1947128683 shutdown_connections 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 --2- 172.21.15.26:0/1947128683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55fc070890 0x7f55fc070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 --2- 172.21.15.26:0/1947128683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc071e40 0x7f55fc070350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 -- 172.21.15.26:0/1947128683 >> 172.21.15.26:0/1947128683 conn(0x7f55fc0f8590 msgr2=0x7f55fc0fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 -- 172.21.15.26:0/1947128683 shutdown_connections 2024-04-16T07:21:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.578+0000 7f5603400700 1 -- 172.21.15.26:0/1947128683 wait complete. 2024-04-16T07:21:28.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 Processor -- start 2024-04-16T07:21:28.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 -- start start 2024-04-16T07:21:28.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55fc071e40 0x7f55fc1145b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc114bb0 con 0x7f55fc070890 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.579+0000 7f5603400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc115400 con 0x7f55fc071e40 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51160/0 (socket says 172.21.15.26:51160) 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 -- 172.21.15.26:0/2780843592 learned_addr learned my addr 172.21.15.26:0/2780843592 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 -- 172.21.15.26:0/2780843592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55fc071e40 msgr2=0x7f55fc1145b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55fc071e40 0x7f55fc1145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.580+0000 7f5600a00700 1 -- 172.21.15.26:0/2780843592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec005040 con 0x7f55fc070890 2024-04-16T07:21:28.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f5600a00700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00b970 tx=0x7f55ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ec00e040 con 0x7f55fc070890 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55ec004290 con 0x7f55fc070890 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ec01c630 con 0x7f55fc070890 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc115680 con 0x7f55fc070890 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.581+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc115bc0 con 0x7f55fc070890 2024-04-16T07:21:28.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.583+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc104040 con 0x7f55fc070890 2024-04-16T07:21:28.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.584+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f55ec00bbf0 con 0x7f55fc070890 2024-04-16T07:21:28.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.585+0000 7f55f8c00700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 0x7f55dc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:28.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.585+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f55ec088570 con 0x7f55fc070890 2024-04-16T07:21:28.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.585+0000 7f55fbe00700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 0x7f55dc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:28.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.586+0000 7f55fbe00700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 0x7f55dc06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f55e4004d00 tx=0x7f55e4009ce0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:28.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.594+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f55ec0582c0 con 0x7f55fc070890 2024-04-16T07:21:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:28 smithi026 ceph-mon[28805]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:28 smithi104 ceph-mon[36163]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:28.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.925+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55fc115ef0 con 0x7f55dc0692b0 2024-04-16T07:21:28.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.926+0000 7f55f8c00700 1 -- 172.21.15.26:0/2780843592 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f55fc115ef0 con 0x7f55dc0692b0 2024-04-16T07:21:28.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 msgr2=0x7f55dc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 0x7f55dc06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f55e4004d00 tx=0x7f55e4009ce0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 msgr2=0x7f55fc114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:28.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00b970 tx=0x7f55ec009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 shutdown_connections 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f55dc0692b0 0x7f55dc06b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55fc070890 0x7f55fc114070 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 --2- 172.21.15.26:0/2780843592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55fc071e40 0x7f55fc1145b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.929+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 >> 172.21.15.26:0/2780843592 conn(0x7f55fc0f8590 msgr2=0x7f55fc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.930+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 shutdown_connections 2024-04-16T07:21:28.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:28.930+0000 7f5603400700 1 -- 172.21.15.26:0/2780843592 wait complete. 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:28.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:28.987+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 -- 172.21.15.26:0/2793610212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e0071f50 msgr2=0x7f02e0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 --2- 172.21.15.26:0/2793610212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e0071f50 0x7f02e0070460 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f02d0004940 tx=0x7f02d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 -- 172.21.15.26:0/2793610212 shutdown_connections 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 --2- 172.21.15.26:0/2793610212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e00709a0 0x7f02e0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 --2- 172.21.15.26:0/2793610212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e0071f50 0x7f02e0070460 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 -- 172.21.15.26:0/2793610212 >> 172.21.15.26:0/2793610212 conn(0x7f02e00f8530 msgr2=0x7f02e00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 -- 172.21.15.26:0/2793610212 shutdown_connections 2024-04-16T07:21:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.088+0000 7f02e7000700 1 -- 172.21.15.26:0/2793610212 wait complete. 2024-04-16T07:21:29.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.089+0000 7f02e7000700 1 Processor -- start 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.089+0000 7f02e7000700 1 -- start start 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02e7000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02e7000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e0071f50 0x7f02e0110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02e7000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e0110800 con 0x7f02e00709a0 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02e7000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e0110ff0 con 0x7f02e0071f50 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02dfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02dfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51182/0 (socket says 172.21.15.26:51182) 2024-04-16T07:21:29.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02dfe00700 1 -- 172.21.15.26:0/1700727915 learned_addr learned my addr 172.21.15.26:0/1700727915 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.090+0000 7f02df400700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e0071f50 0x7f02e0110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02dfe00700 1 -- 172.21.15.26:0/1700727915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e0071f50 msgr2=0x7f02e0110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02dfe00700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e0071f50 0x7f02e0110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02dfe00700 1 -- 172.21.15.26:0/1700727915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0005040 con 0x7f02e00709a0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02dfe00700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f02d000b970 tx=0x7f02d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d000e040 con 0x7f02e00709a0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02d0004290 con 0x7f02e00709a0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e0111270 con 0x7f02e00709a0 2024-04-16T07:21:29.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.091+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d001c630 con 0x7f02e00709a0 2024-04-16T07:21:29.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.092+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e0111830 con 0x7f02e00709a0 2024-04-16T07:21:29.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.093+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e01095a0 con 0x7f02e00709a0 2024-04-16T07:21:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.095+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f02d000bbf0 con 0x7f02e00709a0 2024-04-16T07:21:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.096+0000 7f02d7e00700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 0x7f02c406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.096+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f02d0088690 con 0x7f02e00709a0 2024-04-16T07:21:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.096+0000 7f02df400700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 0x7f02c406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.097+0000 7f02df400700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 0x7f02c406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006fd0 tx=0x7f02cc008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:29.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.101+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f02d0058360 con 0x7f02e00709a0 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:29.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:29.279+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:29.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.332+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f02e0111bd0 con 0x7f02c40692b0 2024-04-16T07:21:29.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.361+0000 7f02d7e00700 1 -- 172.21.15.26:0/1700727915 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f02e0111bd0 con 0x7f02c40692b0 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (6m) 74s ago 8m 13.1M - 0.25.0 c8568f914cd2 d5f7c216b68d 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (9m) 74s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (7m) 81s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (6m) 74s ago 8m 73.0M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (106s) 74s ago 105s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (97s) 74s ago 97s 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (102s) 81s ago 102s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (93s) 81s ago 93s 10.4M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (10m) 74s ago 10m 481M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:21:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (7m) 81s ago 7m 432M - 17.2.7-847-g45317f95 b333581ccdf5 275c78dfc5a9 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (10m) 74s ago 10m 44.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (7m) 81s ago 7m 31.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (9m) 74s ago 9m 8074k - 1.5.0 0da6a335fe13 5198001b496e 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 81s ago 7m 7503k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (6m) 74s ago 6m 43.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (5m) 74s ago 5m 46.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (5m) 74s ago 5m 44.1M 4290M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (4m) 81s ago 4m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (4m) 81s ago 4m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (3m) 81s ago 3m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:21:29.361 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (6m) 74s ago 8m 46.4M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:21:29.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 msgr2=0x7f02c406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 0x7f02c406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006fd0 tx=0x7f02cc008040 comp rx=0 tx=0).stop 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 msgr2=0x7f02e010fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f02d000b970 tx=0x7f02d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 shutdown_connections 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f02c40692b0 0x7f02c406b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02e00709a0 0x7f02e010fc30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 --2- 172.21.15.26:0/1700727915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02e0071f50 0x7f02e0110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 >> 172.21.15.26:0/1700727915 conn(0x7f02e00f8530 msgr2=0x7f02e0102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:29.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 shutdown_connections 2024-04-16T07:21:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.364+0000 7f02e7000700 1 -- 172.21.15.26:0/1700727915 wait complete. 2024-04-16T07:21:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 -- 172.21.15.26:0/1953088192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 msgr2=0x7f7b80070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 --2- 172.21.15.26:0/1953088192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b80070350 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b70004940 tx=0x7f7b70009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 -- 172.21.15.26:0/1953088192 shutdown_connections 2024-04-16T07:21:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 --2- 172.21.15.26:0/1953088192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 0x7f7b80070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 --2- 172.21.15.26:0/1953088192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b80070350 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 -- 172.21.15.26:0/1953088192 >> 172.21.15.26:0/1953088192 conn(0x7f7b800f8390 msgr2=0x7f7b800fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.524+0000 7f7b85400700 1 -- 172.21.15.26:0/1953088192 shutdown_connections 2024-04-16T07:21:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.525+0000 7f7b85400700 1 -- 172.21.15.26:0/1953088192 wait complete. 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.525+0000 7f7b85400700 1 Processor -- start 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.525+0000 7f7b85400700 1 -- start start 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b85400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 0x7f7b8010fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b85400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b85400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b801105d0 con 0x7f7b80071e40 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b85400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80110e20 con 0x7f7b80070890 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51206/0 (socket says 172.21.15.26:51206) 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b7f400700 1 -- 172.21.15.26:0/552751747 learned_addr learned my addr 172.21.15.26:0/552751747 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.526+0000 7f7b7fe00700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 0x7f7b8010fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b7f400700 1 -- 172.21.15.26:0/552751747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 msgr2=0x7f7b8010fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b7f400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 0x7f7b8010fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b7f400700 1 -- 172.21.15.26:0/552751747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70005040 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b7f400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c8c0 tx=0x7f7b6c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6c00a760 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b6c00ad90 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6c0113a0 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.527+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b80111100 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.528+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b80111640 con 0x7f7b80071e40 2024-04-16T07:21:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.529+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8005f5f0 con 0x7f7b80071e40 2024-04-16T07:21:29.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.530+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7b6c012400 con 0x7f7b80071e40 2024-04-16T07:21:29.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.531+0000 7f7b77e00700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 0x7f7b6406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:29.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.531+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7b6c087240 con 0x7f7b80071e40 2024-04-16T07:21:29.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.531+0000 7f7b7fe00700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 0x7f7b6406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:29.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.532+0000 7f7b7fe00700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 0x7f7b6406b760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b3b0 tx=0x7f7b70009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:29.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.535+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7b6c056f90 con 0x7f7b80071e40 2024-04-16T07:21:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:29 smithi026 ceph-mon[28805]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:29.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:29 smithi104 ceph-mon[36163]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:29.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.841+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b8004bed0 con 0x7f7b80071e40 2024-04-16T07:21:29.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.842+0000 7f7b77e00700 1 -- 172.21.15.26:0/552751747 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f7b6c056b20 con 0x7f7b80071e40 2024-04-16T07:21:29.841 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:21:29.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:21:29.841 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:21:29.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:21:29.842 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.845+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 msgr2=0x7f7b6406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.845+0000 7f7b85400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 0x7f7b6406b760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b3b0 tx=0x7f7b70009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 msgr2=0x7f7b8010ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c8c0 tx=0x7f7b6c0044c0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 shutdown_connections 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7b640692b0 0x7f7b6406b760 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b80070890 0x7f7b8010fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 --2- 172.21.15.26:0/552751747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b80071e40 0x7f7b8010ffd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 >> 172.21.15.26:0/552751747 conn(0x7f7b800f8390 msgr2=0x7f7b800f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 shutdown_connections 2024-04-16T07:21:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.846+0000 7f7b85400700 1 -- 172.21.15.26:0/552751747 wait complete. 2024-04-16T07:21:29.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:29.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:29.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:29.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:29.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:29.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:29.987+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:29.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 -- 172.21.15.26:0/871907043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc0708f0 msgr2=0x7f0ccc070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:29.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/871907043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc070d60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8004940 tx=0x7f0cb8009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:29.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 -- 172.21.15.26:0/871907043 shutdown_connections 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/871907043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc070d60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/871907043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:29.999+0000 7f0cd2c00700 1 -- 172.21.15.26:0/871907043 >> 172.21.15.26:0/871907043 conn(0x7f0ccc0f8530 msgr2=0x7f0ccc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.000+0000 7f0cd2c00700 1 -- 172.21.15.26:0/871907043 shutdown_connections 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.000+0000 7f0cd2c00700 1 -- 172.21.15.26:0/871907043 wait complete. 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.000+0000 7f0cd2c00700 1 Processor -- start 2024-04-16T07:21:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0cd2c00700 1 -- start start 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0cd2c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0cd2c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0cd2c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc114b50 con 0x7f0ccc071ea0 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0cd2c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc1153a0 con 0x7f0ccc0708f0 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.001+0000 7f0ccb400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51226/0 (socket says 172.21.15.26:51226) 2024-04-16T07:21:30.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 -- 172.21.15.26:0/1103353405 learned_addr learned my addr 172.21.15.26:0/1103353405 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:30.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccbe00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 -- 172.21.15.26:0/1103353405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0708f0 msgr2=0x7f0ccc114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 -- 172.21.15.26:0/1103353405 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8005040 con 0x7f0ccc071ea0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.002+0000 7f0ccb400700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009150 tx=0x7f0cb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.003+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cb800e040 con 0x7f0ccc071ea0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.003+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cb800bbf0 con 0x7f0ccc071ea0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.003+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cb801c630 con 0x7f0ccc071ea0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.003+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc115620 con 0x7f0ccc071ea0 2024-04-16T07:21:30.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.003+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc115b60 con 0x7f0ccc071ea0 2024-04-16T07:21:30.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.004+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc05f5f0 con 0x7f0ccc071ea0 2024-04-16T07:21:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.006+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0cb800bd60 con 0x7f0ccc071ea0 2024-04-16T07:21:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.007+0000 7f0cc3e00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 0x7f0cb006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.007+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0cb8088550 con 0x7f0ccc071ea0 2024-04-16T07:21:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.007+0000 7f0ccbe00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 0x7f0cb006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.008+0000 7f0ccbe00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 0x7f0cb006b760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007fd0 tx=0x7f0cbc009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.012+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0cb8058410 con 0x7f0ccc071ea0 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:30.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:30.279+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.299+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0ccc10bef0 con 0x7f0ccc071ea0 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.300+0000 7f0cc3e00700 1 -- 172.21.15.26:0/1103353405 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f0cb8057fa0 con 0x7f0ccc071ea0 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:21:30.299 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:21:30.300 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:21:30.301 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:21:30.302 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:21:30.302 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:21:30.302 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:21:30.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 msgr2=0x7f0cb006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 0x7f0cb006b760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007fd0 tx=0x7f0cbc009040 comp rx=0 tx=0).stop 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 msgr2=0x7f0ccc114550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009150 tx=0x7f0cb8009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 shutdown_connections 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0cb00692b0 0x7f0cb006b760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0708f0 0x7f0ccc114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 --2- 172.21.15.26:0/1103353405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ccc071ea0 0x7f0ccc114550 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 >> 172.21.15.26:0/1103353405 conn(0x7f0ccc0f8530 msgr2=0x7f0ccc0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 shutdown_connections 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.303+0000 7f0cd2c00700 1 -- 172.21.15.26:0/1103353405 wait complete. 2024-04-16T07:21:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 -- 172.21.15.26:0/683417868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 msgr2=0x7f0d64070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 --2- 172.21.15.26:0/683417868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64070350 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0d58004940 tx=0x7f0d58009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 -- 172.21.15.26:0/683417868 shutdown_connections 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 --2- 172.21.15.26:0/683417868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d64070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 --2- 172.21.15.26:0/683417868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64070350 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 -- 172.21.15.26:0/683417868 >> 172.21.15.26:0/683417868 conn(0x7f0d640f8590 msgr2=0x7f0d640fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 -- 172.21.15.26:0/683417868 shutdown_connections 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.461+0000 7f0d6c000700 1 -- 172.21.15.26:0/683417868 wait complete. 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 Processor -- start 2024-04-16T07:21:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 -- start start 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d6410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d64110850 con 0x7f0d64071e40 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.462+0000 7f0d6c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d64110f90 con 0x7f0d64070890 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51244/0 (socket says 172.21.15.26:51244) 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 -- 172.21.15.26:0/3362865078 learned_addr learned my addr 172.21.15.26:0/3362865078 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d69600700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d6410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 -- 172.21.15.26:0/3362865078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 msgr2=0x7f0d6410fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d6410fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d68c00700 1 -- 172.21.15.26:0/3362865078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d58005040 con 0x7f0d64071e40 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.463+0000 7f0d69600700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d6410fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.464+0000 7f0d68c00700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d50002a50 tx=0x7f0d500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.464+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d50007cc0 con 0x7f0d64071e40 2024-04-16T07:21:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.464+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d5000f040 con 0x7f0d64071e40 2024-04-16T07:21:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.464+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d64111270 con 0x7f0d64071e40 2024-04-16T07:21:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.464+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5000db70 con 0x7f0d64071e40 2024-04-16T07:21:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.465+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d64111830 con 0x7f0d64071e40 2024-04-16T07:21:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.466+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6405f5f0 con 0x7f0d64071e40 2024-04-16T07:21:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.470+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0d5001c400 con 0x7f0d64071e40 2024-04-16T07:21:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.472+0000 7f0d61600700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 0x7f0d4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.472+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d50086990 con 0x7f0d64071e40 2024-04-16T07:21:30.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.472+0000 7f0d69600700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 0x7f0d4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.473+0000 7f0d69600700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 0x7f0d4c06b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b3b0 tx=0x7f0d58009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.476+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0d50056660 con 0x7f0d64071e40 2024-04-16T07:21:30.638 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:30 smithi026 ceph-mon[28805]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:30.638 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:30 smithi026 ceph-mon[28805]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:30.638 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:30 smithi026 ceph-mon[28805]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:30.638 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:30 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/552751747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:21:30.638 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:30 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1103353405' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:21:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.726+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d64111c20 con 0x7f0d4c0692b0 2024-04-16T07:21:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.727+0000 7f0d61600700 1 -- 172.21.15.26:0/3362865078 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f0d64111c20 con 0x7f0d4c0692b0 2024-04-16T07:21:30.726 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:21:30.726 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-16T07:21:30.726 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "", 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 image", 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:21:30.727 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:21:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 msgr2=0x7f0d4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 0x7f0d4c06b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b3b0 tx=0x7f0d58009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 msgr2=0x7f0d64110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d50002a50 tx=0x7f0d500046b0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 shutdown_connections 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d4c0692b0 0x7f0d4c06b760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d64070890 0x7f0d6410fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 --2- 172.21.15.26:0/3362865078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d64071e40 0x7f0d64110250 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.730+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 >> 172.21.15.26:0/3362865078 conn(0x7f0d640f8590 msgr2=0x7f0d640f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.731+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 shutdown_connections 2024-04-16T07:21:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.731+0000 7f0d6c000700 1 -- 172.21.15.26:0/3362865078 wait complete. 2024-04-16T07:21:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:30 smithi104 ceph-mon[36163]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:30 smithi104 ceph-mon[36163]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:30 smithi104 ceph-mon[36163]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:30.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:30 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/552751747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:21:30.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:30 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1103353405' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.895+0000 7f6bfe400700 1 -- 172.21.15.26:0/1838227869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf81046d0 msgr2=0x7f6bf8104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.895+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1838227869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf81046d0 0x7f6bf8104b40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6be0004940 tx=0x7f6be0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 -- 172.21.15.26:0/1838227869 shutdown_connections 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1838227869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf81046d0 0x7f6bf8104b40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1838227869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf8103420 0x7f6bf8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 -- 172.21.15.26:0/1838227869 >> 172.21.15.26:0/1838227869 conn(0x7f6bf8072640 msgr2=0x7f6bf8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 -- 172.21.15.26:0/1838227869 shutdown_connections 2024-04-16T07:21:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.896+0000 7f6bfe400700 1 -- 172.21.15.26:0/1838227869 wait complete. 2024-04-16T07:21:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 Processor -- start 2024-04-16T07:21:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 -- start start 2024-04-16T07:21:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf81046d0 0x7f6bf8114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8114a90 con 0x7f6bf8103420 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bfe400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8114c00 con 0x7f6bf81046d0 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bf7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bf7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51250/0 (socket says 172.21.15.26:51250) 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.897+0000 7f6bf7400700 1 -- 172.21.15.26:0/1433619522 learned_addr learned my addr 172.21.15.26:0/1433619522 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6befe00700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf81046d0 0x7f6bf8114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bf7400700 1 -- 172.21.15.26:0/1433619522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf81046d0 msgr2=0x7f6bf8114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bf7400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf81046d0 0x7f6bf8114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bf7400700 1 -- 172.21.15.26:0/1433619522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0005040 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bf7400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c830 tx=0x7f6be8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6be800a760 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6be800ad90 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6be80113a0 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.898+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf8119660 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.899+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf8119c20 con 0x7f6bf8103420 2024-04-16T07:21:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.900+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf804c710 con 0x7f6bf8103420 2024-04-16T07:21:30.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.901+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6be8012400 con 0x7f6bf8103420 2024-04-16T07:21:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.902+0000 7f6bef400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 0x7f6bdc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:21:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.902+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6be8087260 con 0x7f6bf8103420 2024-04-16T07:21:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.902+0000 7f6befe00700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 0x7f6bdc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:21:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.903+0000 7f6befe00700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 0x7f6bdc06b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6be000b970 tx=0x7f6be0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:21:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:30.906+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6be8056fb0 con 0x7f6bf8103420 2024-04-16T07:21:30.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.987+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:30.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.987+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:30.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.987+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:30.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.987+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:30.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.988+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:30.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:30.988+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:31.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.206+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6bf811a140 con 0x7f6bf8103420 2024-04-16T07:21:31.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.206+0000 7f6bef400700 1 -- 172.21.15.26:0/1433619522 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6be805c070 con 0x7f6bf8103420 2024-04-16T07:21:31.205 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 msgr2=0x7f6bdc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 0x7f6bdc06b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6be000b970 tx=0x7f6be0009e30 comp rx=0 tx=0).stop 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 msgr2=0x7f6bf8113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c830 tx=0x7f6be8004470 comp rx=0 tx=0).stop 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 shutdown_connections 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f6bdc0692a0 0x7f6bdc06b750 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf8103420 0x7f6bf8113f50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 --2- 172.21.15.26:0/1433619522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bf81046d0 0x7f6bf8114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 >> 172.21.15.26:0/1433619522 conn(0x7f6bf8072640 msgr2=0x7f6bf8073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 shutdown_connections 2024-04-16T07:21:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:21:31.209+0000 7f6bfe400700 1 -- 172.21.15.26:0/1433619522 wait complete. 2024-04-16T07:21:31.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:31.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:31.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:31.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:31.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:31.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:31.279+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:31 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1433619522' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:21:31.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:31 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:31 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1433619522' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:21:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:31 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:31.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:31.988+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:32.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:32.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 5) v5 -- 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 9 ==== client_session(renewcaps seq 5) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004270 con 0x55fc3f51baa0 2024-04-16T07:21:32.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:32.280+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 5) v1 from mds.0 2024-04-16T07:21:32.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:32 smithi026 ceph-mon[28805]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:32.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:32 smithi026 ceph-mon[28805]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:21:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:21:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:32 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:21:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:32 smithi104 ceph-mon[36163]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:21:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:32 smithi104 ceph-mon[36163]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:21:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:21:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:32 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:32.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:32.989+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:33.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:33.280+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:33 smithi026 ceph-mon[28805]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T07:21:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:33 smithi026 ceph-mon[28805]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T07:21:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:33 smithi026 ceph-mon[28805]: Upgrade: Need to upgrade myself (mgr.smithi026.tawexg) 2024-04-16T07:21:33.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:33 smithi026 ceph-mon[28805]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi104 2024-04-16T07:21:33.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:33 smithi104 ceph-mon[36163]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T07:21:33.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:33 smithi104 ceph-mon[36163]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T07:21:33.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:33 smithi104 ceph-mon[36163]: Upgrade: Need to upgrade myself (mgr.smithi026.tawexg) 2024-04-16T07:21:33.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:33 smithi104 ceph-mon[36163]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi104 2024-04-16T07:21:33.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:33.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:33.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:33.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:33.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:33.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:33.988+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:34.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:34.280+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:34.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:34 smithi026 ceph-mon[28805]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:34 smithi104 ceph-mon[36163]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:34.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:34.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:34.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:34.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:34.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:34.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:34.989+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:35.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:35.280+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:35.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:35.989+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:36.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.280+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:36.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:36.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.280+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:36.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.280+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:36.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.281+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:36.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:36.281+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:36 smithi104 ceph-mon[36163]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:36.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:36.990+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:36 smithi026 ceph-mon[28805]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:37.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:37.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:37.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:37.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:37.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:37.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:37.281+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:37.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:37.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:37.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:37.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:37.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:37.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:37.990+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:38.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:38.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:38.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:38.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:38.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:38.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:38.281+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:38 smithi104 ceph-mon[36163]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:38.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:38.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:38.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:38.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:38.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:38.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:38.990+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:38 smithi026 ceph-mon[28805]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:39.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:39.281+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:39.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:39.991+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:40.281+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:40 smithi104 ceph-mon[36163]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:40.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:40.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:40.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:40.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:40.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:40.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:40.991+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:41.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:40 smithi026 ceph-mon[28805]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:41.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.281+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:41.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:41.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.281+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:41.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.281+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:41.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.282+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:41.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:41.282+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:41.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:41.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:41.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:41.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:41.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:41.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:41.991+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:42.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:42.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:42.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:42.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:42.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:42.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:42.282+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:42 smithi104 ceph-mon[36163]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:42.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:42.991+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:42 smithi026 ceph-mon[28805]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:43.282+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 6) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:43.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:43.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:43.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:43.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:43.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 10 ==== client_session(renewcaps seq 6) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68001d80 con 0x563ccaa1cfa0 2024-04-16T07:21:43.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:43.992+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 6) v1 from mds.0 2024-04-16T07:21:44.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:44.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:44.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:44.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:44.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:44.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:44.282+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:44 smithi104 ceph-mon[36163]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:44.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:44.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:44.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:44.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:44.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:44.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:44.991+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:45.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:44 smithi026 ceph-mon[28805]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:45.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.282+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:45.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:45.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.282+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:45.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.282+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:45.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.283+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:45.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:45.283+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:45.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:45.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:45.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:45.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:45.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:45.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:45.992+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:46.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:46.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:46.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:46.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:46.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:46.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:46.283+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:46 smithi104 ceph-mon[36163]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:46 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:46.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:46.992+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:47.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:46 smithi026 ceph-mon[28805]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:47.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:46 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:47.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:47.283+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:47.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:47.993+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:48.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:48.283+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:48.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:48 smithi104 ceph-mon[36163]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:48.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:48.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:48.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:48.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:48.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:48.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:48.993+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:49.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:48 smithi026 ceph-mon[28805]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:49.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.283+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:49.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:49.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.283+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:49.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.283+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:49.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.284+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:49.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:49.284+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:49.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:49.993+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:50.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:50.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:50.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:50.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:50.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:50.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:50.284+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:50 smithi104 ceph-mon[36163]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:50.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:50.994+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:51.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:50 smithi026 ceph-mon[28805]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:51.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:51.284+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:51.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:51.994+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 6) v5 -- 0x7fe9ac004610 con 0x55fc3f51baa0 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:52.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:52.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:52.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:52.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.284+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:52.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.285+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 10 ==== client_session(renewcaps seq 6) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004610 con 0x55fc3f51baa0 2024-04-16T07:21:52.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:52.285+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 6) v1 from mds.0 2024-04-16T07:21:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:52 smithi104 ceph-mon[36163]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:52.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:52.994+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:53.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:52 smithi026 ceph-mon[28805]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:53.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:53.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:53.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:53.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:53.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:53.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:53.284+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:53.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:53.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:53.995+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:54.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:54.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:54.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:54.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:54.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:54.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:54.285+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:54 smithi104 ceph-mon[36163]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:54.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:54.994+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:55.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:54 smithi026 ceph-mon[28805]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:55.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:55.285+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:55.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:55.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:55.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:55.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:55.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:55.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:55.994+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:56.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:56.285+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:56 smithi104 ceph-mon[36163]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:56.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:56.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:56.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:56.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:56.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:56.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:56.995+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:57.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:56 smithi026 ceph-mon[28805]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:57.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:57.285+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:57.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:57.995+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:58.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.285+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.286+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.286+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:58.286+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:21:58 smithi104 ceph-mon[36163]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:58.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:58.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:58.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:58.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:58.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:58.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:58.995+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:21:58 smithi026 ceph-mon[28805]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:21:59.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:21:59.286+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:21:59.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:21:59.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:21:59.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:21:59.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:21:59.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:21:59.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:21:59.995+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:00.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:00.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:00.286+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:00.996+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:00 smithi026 ceph-mon[28805]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:00 smithi104 ceph-mon[36163]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:01.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:01.286+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.383+0000 7f038ce00700 1 -- 172.21.15.26:0/2692511115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 msgr2=0x7f03880fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.383+0000 7f038ce00700 1 --2- 172.21.15.26:0/2692511115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f03880fd290 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0378004940 tx=0x7f0378009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 -- 172.21.15.26:0/2692511115 shutdown_connections 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 --2- 172.21.15.26:0/2692511115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 0x7f03880fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 --2- 172.21.15.26:0/2692511115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f03880fd290 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 -- 172.21.15.26:0/2692511115 >> 172.21.15.26:0/2692511115 conn(0x7f03880f8580 msgr2=0x7f03880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 -- 172.21.15.26:0/2692511115 shutdown_connections 2024-04-16T07:22:01.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.384+0000 7f038ce00700 1 -- 172.21.15.26:0/2692511115 wait complete. 2024-04-16T07:22:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 Processor -- start 2024-04-16T07:22:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 -- start start 2024-04-16T07:22:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 0x7f0388110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0388110650 con 0x7f03880fce80 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.385+0000 7f038ce00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03881107c0 con 0x7f03880fdff0 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47090/0 (socket says 172.21.15.26:47090) 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 -- 172.21.15.26:0/2803967822 learned_addr learned my addr 172.21.15.26:0/2803967822 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385400700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 0x7f0388110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 -- 172.21.15.26:0/2803967822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 msgr2=0x7f0388110050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 0x7f0388110050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 -- 172.21.15.26:0/2803967822 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0378005040 con 0x7f03880fce80 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.386+0000 7f0385e00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0378009150 tx=0x7f0378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.387+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037800e040 con 0x7f03880fce80 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.387+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f037800bbf0 con 0x7f03880fce80 2024-04-16T07:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.387+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03881151c0 con 0x7f03880fce80 2024-04-16T07:22:01.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.387+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037801c630 con 0x7f03880fce80 2024-04-16T07:22:01.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.387+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0388115780 con 0x7f03880fce80 2024-04-16T07:22:01.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.388+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0388101980 con 0x7f03880fce80 2024-04-16T07:22:01.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.389+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f037800bd60 con 0x7f03880fce80 2024-04-16T07:22:01.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.390+0000 7f037e000700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 0x7f036c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.390+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0378088610 con 0x7f03880fce80 2024-04-16T07:22:01.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.391+0000 7f0385400700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 0x7f036c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.391+0000 7f0385400700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 0x7f036c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0374009fd0 tx=0x7f0374009f60 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:01.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.395+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0378058410 con 0x7f03880fce80 2024-04-16T07:22:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.633+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f038802b2f0 con 0x7f036c0692a0 2024-04-16T07:22:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.635+0000 7f037e000700 1 -- 172.21.15.26:0/2803967822 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f038802b2f0 con 0x7f036c0692a0 2024-04-16T07:22:01.634 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:01 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 msgr2=0x7f036c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 0x7f036c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0374009fd0 tx=0x7f0374009f60 comp rx=0 tx=0).stop 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 msgr2=0x7f038810fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0378009150 tx=0x7f0378009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 shutdown_connections 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f036c0692a0 0x7f036c06b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.637+0000 7f038ce00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03880fce80 0x7f038810fb10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.638+0000 7f038ce00700 1 --2- 172.21.15.26:0/2803967822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03880fdff0 0x7f0388110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.638+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 >> 172.21.15.26:0/2803967822 conn(0x7f03880f8580 msgr2=0x7f03880f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.638+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 shutdown_connections 2024-04-16T07:22:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.638+0000 7f038ce00700 1 -- 172.21.15.26:0/2803967822 wait complete. 2024-04-16T07:22:01.653 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.799+0000 7f9ac5000700 1 -- 172.21.15.26:0/880820475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0071f50 msgr2=0x7f9ac0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.799+0000 7f9ac5000700 1 --2- 172.21.15.26:0/880820475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0071f50 0x7f9ac0070460 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0004940 tx=0x7f9ab0009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 -- 172.21.15.26:0/880820475 shutdown_connections 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 --2- 172.21.15.26:0/880820475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac00709a0 0x7f9ac0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 --2- 172.21.15.26:0/880820475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0071f50 0x7f9ac0070460 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 -- 172.21.15.26:0/880820475 >> 172.21.15.26:0/880820475 conn(0x7f9ac00f8530 msgr2=0x7f9ac00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 -- 172.21.15.26:0/880820475 shutdown_connections 2024-04-16T07:22:01.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.800+0000 7f9ac5000700 1 -- 172.21.15.26:0/880820475 wait complete. 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 Processor -- start 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 -- start start 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0071f50 0x7f9ac01145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0114bd0 con 0x7f9ac00709a0 2024-04-16T07:22:01.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.801+0000 7f9ac5000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0115420 con 0x7f9ac0071f50 2024-04-16T07:22:01.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47096/0 (socket says 172.21.15.26:47096) 2024-04-16T07:22:01.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 -- 172.21.15.26:0/638117400 learned_addr learned my addr 172.21.15.26:0/638117400 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9ab7e00700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0071f50 0x7f9ac01145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 -- 172.21.15.26:0/638117400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0071f50 msgr2=0x7f9ac01145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0071f50 0x7f9ac01145d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 -- 172.21.15.26:0/638117400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0005040 con 0x7f9ac00709a0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.802+0000 7f9abde00700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000b970 tx=0x7f9ab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.803+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab000e040 con 0x7f9ac00709a0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.803+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ab0004290 con 0x7f9ac00709a0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.803+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab001c630 con 0x7f9ac00709a0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.803+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac01156a0 con 0x7f9ac00709a0 2024-04-16T07:22:01.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.803+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0115be0 con 0x7f9ac00709a0 2024-04-16T07:22:01.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.804+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac005f5f0 con 0x7f9ac00709a0 2024-04-16T07:22:01.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.805+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9ab000bbf0 con 0x7f9ac00709a0 2024-04-16T07:22:01.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.806+0000 7f9ab6000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 0x7f9aa406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:01.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.806+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9ab0088570 con 0x7f9ac00709a0 2024-04-16T07:22:01.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.806+0000 7f9ab7e00700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 0x7f9aa406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:01.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.807+0000 7f9ab7e00700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 0x7f9aa406b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8004990 tx=0x7f9aa8009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:01.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:01.811+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9ab00582c0 con 0x7f9ac00709a0 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:01.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:01.996+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.049+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ac0115f80 con 0x7f9aa40692b0 2024-04-16T07:22:02.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.051+0000 7f9ab6000700 1 -- 172.21.15.26:0/638117400 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f9ac0115f80 con 0x7f9aa40692b0 2024-04-16T07:22:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.054+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 msgr2=0x7f9aa406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.054+0000 7f9ac5000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 0x7f9aa406b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8004990 tx=0x7f9aa8009ee0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 msgr2=0x7f9ac0114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000b970 tx=0x7f9ab0009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 shutdown_connections 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f9aa40692b0 0x7f9aa406b760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac00709a0 0x7f9ac0114090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 --2- 172.21.15.26:0/638117400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0071f50 0x7f9ac01145d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 >> 172.21.15.26:0/638117400 conn(0x7f9ac00f8530 msgr2=0x7f9ac00f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 shutdown_connections 2024-04-16T07:22:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.055+0000 7f9ac5000700 1 -- 172.21.15.26:0/638117400 wait complete. 2024-04-16T07:22:02.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:01 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 -- 172.21.15.26:0/849497923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 msgr2=0x7f396c06ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 --2- 172.21.15.26:0/849497923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c06ee20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3958004940 tx=0x7f3958009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 -- 172.21.15.26:0/849497923 shutdown_connections 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 --2- 172.21.15.26:0/849497923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c06ee20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 --2- 172.21.15.26:0/849497923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 0x7f396c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 -- 172.21.15.26:0/849497923 >> 172.21.15.26:0/849497923 conn(0x7f396c069df0 msgr2=0x7f396c06c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 -- 172.21.15.26:0/849497923 shutdown_connections 2024-04-16T07:22:02.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.218+0000 7f3973200700 1 -- 172.21.15.26:0/849497923 wait complete. 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 Processor -- start 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 -- start start 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 0x7f396c11cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c11d240 con 0x7f396c06e9b0 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.219+0000 7f3973200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c11d3b0 con 0x7f396c06f9c0 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47106/0 (socket says 172.21.15.26:47106) 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 -- 172.21.15.26:0/646856188 learned_addr learned my addr 172.21.15.26:0/646856188 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396b400700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 0x7f396c11cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 -- 172.21.15.26:0/646856188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 msgr2=0x7f396c11cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 0x7f396c11cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 -- 172.21.15.26:0/646856188 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958005040 con 0x7f396c06e9b0 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.220+0000 7f396be00700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c830 tx=0x7f395c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.221+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c00a760 con 0x7f396c06e9b0 2024-04-16T07:22:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.221+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f395c00ad90 con 0x7f396c06e9b0 2024-04-16T07:22:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.221+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c121e10 con 0x7f396c06e9b0 2024-04-16T07:22:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.221+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c0113a0 con 0x7f396c06e9b0 2024-04-16T07:22:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.221+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c122340 con 0x7f396c06e9b0 2024-04-16T07:22:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.222+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c04c710 con 0x7f396c06e9b0 2024-04-16T07:22:02.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.223+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f395c012400 con 0x7f396c06e9b0 2024-04-16T07:22:02.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.224+0000 7f3963e00700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 0x7f395006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.224+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f395c087210 con 0x7f396c06e9b0 2024-04-16T07:22:02.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.224+0000 7f396b400700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 0x7f395006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.225+0000 7f396b400700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 0x7f395006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f395800b970 tx=0x7f3958009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:02.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.229+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f395c056f60 con 0x7f396c06e9b0 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:02.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:02.286+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.454+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f396c122610 con 0x7f39500692a0 2024-04-16T07:22:02.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.484+0000 7f3963e00700 1 -- 172.21.15.26:0/646856188 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f396c122610 con 0x7f39500692a0 2024-04-16T07:22:02.483 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:22:02.483 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (7m) 107s ago 9m 13.1M - 0.25.0 c8568f914cd2 d5f7c216b68d 2024-04-16T07:22:02.483 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (9m) 107s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (8m) 114s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (7m) 107s ago 8m 73.0M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (2m) 107s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (2m) 107s ago 2m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (2m) 114s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (2m) 114s ago 2m 10.4M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (11m) 107s ago 11m 481M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:22:02.484 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (8m) 114s ago 8m 432M - 17.2.7-847-g45317f95 b333581ccdf5 275c78dfc5a9 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (11m) 107s ago 11m 44.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (7m) 114s ago 7m 31.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (9m) 107s ago 9m 8074k - 1.5.0 0da6a335fe13 5198001b496e 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 114s ago 8m 7503k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (7m) 107s ago 7m 43.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (6m) 107s ago 6m 46.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (5m) 107s ago 5m 44.1M 4290M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (5m) 114s ago 5m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (4m) 114s ago 4m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:22:02.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (4m) 114s ago 4m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:22:02.486 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (7m) 107s ago 8m 46.4M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:22:02.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 msgr2=0x7f395006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 0x7f395006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f395800b970 tx=0x7f3958009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:02.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 msgr2=0x7f396c11c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c830 tx=0x7f395c004470 comp rx=0 tx=0).stop 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 shutdown_connections 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f39500692a0 0x7f395006b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f396c06e9b0 0x7f396c11c700 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 --2- 172.21.15.26:0/646856188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c06f9c0 0x7f396c11cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 >> 172.21.15.26:0/646856188 conn(0x7f396c069df0 msgr2=0x7f396c06b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:02.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 shutdown_connections 2024-04-16T07:22:02.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.487+0000 7f3973200700 1 -- 172.21.15.26:0/646856188 wait complete. 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.652+0000 7fa752a00700 1 -- 172.21.15.26:0/2763077093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 msgr2=0x7fa74c0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.652+0000 7fa752a00700 1 --2- 172.21.15.26:0/2763077093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c0fd290 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa73c004940 tx=0x7fa73c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 -- 172.21.15.26:0/2763077093 shutdown_connections 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 --2- 172.21.15.26:0/2763077093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 0x7fa74c0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 --2- 172.21.15.26:0/2763077093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c0fd290 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 -- 172.21.15.26:0/2763077093 >> 172.21.15.26:0/2763077093 conn(0x7fa74c0f8580 msgr2=0x7fa74c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:02.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 -- 172.21.15.26:0/2763077093 shutdown_connections 2024-04-16T07:22:02.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.653+0000 7fa752a00700 1 -- 172.21.15.26:0/2763077093 wait complete. 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.654+0000 7fa752a00700 1 Processor -- start 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.654+0000 7fa752a00700 1 -- start start 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.654+0000 7fa752a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.654+0000 7fa752a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 0x7fa74c110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa752a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c110680 con 0x7fa74c0fce80 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa752a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1107f0 con 0x7fa74c0fdff0 2024-04-16T07:22:02.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47132/0 (socket says 172.21.15.26:47132) 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 -- 172.21.15.26:0/3949576382 learned_addr learned my addr 172.21.15.26:0/3949576382 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74b400700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 0x7fa74c110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 -- 172.21.15.26:0/3949576382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 msgr2=0x7fa74c110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 0x7fa74c110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.655+0000 7fa74be00700 1 -- 172.21.15.26:0/3949576382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c005040 con 0x7fa74c0fce80 2024-04-16T07:22:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa74be00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009150 tx=0x7fa73c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73c00e040 con 0x7fa74c0fce80 2024-04-16T07:22:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa73c00bcc0 con 0x7fa74c0fce80 2024-04-16T07:22:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c1151f0 con 0x7fa74c0fce80 2024-04-16T07:22:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73c01c630 con 0x7fa74c0fce80 2024-04-16T07:22:02.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.656+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c1157b0 con 0x7fa74c0fce80 2024-04-16T07:22:02.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.658+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c101980 con 0x7fa74c0fce80 2024-04-16T07:22:02.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.659+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa73c013420 con 0x7fa74c0fce80 2024-04-16T07:22:02.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.660+0000 7fa743e00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 0x7fa73006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:02.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.660+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa73c087f90 con 0x7fa74c0fce80 2024-04-16T07:22:02.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.660+0000 7fa74b400700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 0x7fa73006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.661+0000 7fa74b400700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 0x7fa73006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa738007fd0 tx=0x7fa738009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.665+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa73c057ce0 con 0x7fa74c0fce80 2024-04-16T07:22:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:02 smithi026 ceph-mon[28805]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.968+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa74c1058e0 con 0x7fa74c0fce80 2024-04-16T07:22:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.969+0000 7fa743e00700 1 -- 172.21.15.26:0/3949576382 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa73c057870 con 0x7fa74c0fce80 2024-04-16T07:22:02.968 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:22:02.968 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:22:02.968 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:22:02.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:22:02.969 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 msgr2=0x7fa73006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 0x7fa73006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa738007fd0 tx=0x7fa738009040 comp rx=0 tx=0).stop 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 msgr2=0x7fa74c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009150 tx=0x7fa73c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 shutdown_connections 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fa7300692a0 0x7fa73006b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa74c0fce80 0x7fa74c10fb40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 --2- 172.21.15.26:0/3949576382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa74c0fdff0 0x7fa74c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 >> 172.21.15.26:0/3949576382 conn(0x7fa74c0f8580 msgr2=0x7fa74c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.972+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 shutdown_connections 2024-04-16T07:22:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:02.973+0000 7fa752a00700 1 -- 172.21.15.26:0/3949576382 wait complete. 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:02.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:02.996+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:02 smithi104 ceph-mon[36163]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 -- 172.21.15.26:0/3678666463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 msgr2=0x7f0d3c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 --2- 172.21.15.26:0/3678666463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0ffa30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0d30004940 tx=0x7f0d30009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 -- 172.21.15.26:0/3678666463 shutdown_connections 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 --2- 172.21.15.26:0/3678666463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 0x7f0d3c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 --2- 172.21.15.26:0/3678666463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0ffa30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.130+0000 7f0d44200700 1 -- 172.21.15.26:0/3678666463 >> 172.21.15.26:0/3678666463 conn(0x7f0d3c0f74c0 msgr2=0x7f0d3c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.131+0000 7f0d44200700 1 -- 172.21.15.26:0/3678666463 shutdown_connections 2024-04-16T07:22:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.131+0000 7f0d44200700 1 -- 172.21.15.26:0/3678666463 wait complete. 2024-04-16T07:22:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.131+0000 7f0d44200700 1 Processor -- start 2024-04-16T07:22:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d44200700 1 -- start start 2024-04-16T07:22:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d44200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d44200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 0x7f0d3c0fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d44200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c0fbec0 con 0x7f0d3c0662f0 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d44200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c0fc000 con 0x7f0d3c0fff70 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d41800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d41800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47148/0 (socket says 172.21.15.26:47148) 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.132+0000 7f0d41800700 1 -- 172.21.15.26:0/4248443577 learned_addr learned my addr 172.21.15.26:0/4248443577 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d40e00700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 0x7f0d3c0fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d41800700 1 -- 172.21.15.26:0/4248443577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 msgr2=0x7f0d3c0fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d41800700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 0x7f0d3c0fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d41800700 1 -- 172.21.15.26:0/4248443577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d30005040 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d41800700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000b970 tx=0x7f0d30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3000e040 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d30004290 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.133+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3001c630 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.134+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c0fc250 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.134+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c115a90 con 0x7f0d3c0662f0 2024-04-16T07:22:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.135+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c05f5f0 con 0x7f0d3c0662f0 2024-04-16T07:22:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.136+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0d3000bbf0 con 0x7f0d3c0662f0 2024-04-16T07:22:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.137+0000 7f0d39600700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 0x7f0d2406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.137+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d300884b0 con 0x7f0d3c0662f0 2024-04-16T07:22:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.137+0000 7f0d40e00700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 0x7f0d2406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.138+0000 7f0d40e00700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 0x7f0d2406b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d28004990 tx=0x7f0d28009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.141+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0d30054830 con 0x7f0d3c0662f0 2024-04-16T07:22:03.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.286+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:03.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:03.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.286+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:03.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.287+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:03.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.287+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:03.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:03.287+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:03.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.398+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0d3c115c20 con 0x7f0d3c0662f0 2024-04-16T07:22:03.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.398+0000 7f0d39600700 1 -- 172.21.15.26:0/4248443577 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f0d30013e60 con 0x7f0d3c0662f0 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:22:03.398 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:22:03.399 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:03.400 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:03.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 msgr2=0x7f0d2406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 0x7f0d2406b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d28004990 tx=0x7f0d28009ee0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 msgr2=0x7f0d3c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000b970 tx=0x7f0d30009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 shutdown_connections 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f0d240692b0 0x7f0d2406b760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d3c0662f0 0x7f0d3c0fd3f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 --2- 172.21.15.26:0/4248443577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d3c0fff70 0x7f0d3c0fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.402+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 >> 172.21.15.26:0/4248443577 conn(0x7f0d3c0f74c0 msgr2=0x7f0d3c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.403+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 shutdown_connections 2024-04-16T07:22:03.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.403+0000 7f0d44200700 1 -- 172.21.15.26:0/4248443577 wait complete. 2024-04-16T07:22:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:22:03.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 -- 172.21.15.26:0/1154246941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c1046d0 msgr2=0x7fb30c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 --2- 172.21.15.26:0/1154246941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c1046d0 0x7fb30c104b40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8004940 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 -- 172.21.15.26:0/1154246941 shutdown_connections 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 --2- 172.21.15.26:0/1154246941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c1046d0 0x7fb30c104b40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 --2- 172.21.15.26:0/1154246941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c103420 0x7fb30c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.562+0000 7fb312e00700 1 -- 172.21.15.26:0/1154246941 >> 172.21.15.26:0/1154246941 conn(0x7fb30c072640 msgr2=0x7fb30c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.563+0000 7fb312e00700 1 -- 172.21.15.26:0/1154246941 shutdown_connections 2024-04-16T07:22:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.563+0000 7fb312e00700 1 -- 172.21.15.26:0/1154246941 wait complete. 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 Processor -- start 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 -- start start 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c1046d0 0x7fb30c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c114a90 con 0x7fb30c103420 2024-04-16T07:22:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.564+0000 7fb312e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c114c00 con 0x7fb30c1046d0 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47172/0 (socket says 172.21.15.26:47172) 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 -- 172.21.15.26:0/2653701099 learned_addr learned my addr 172.21.15.26:0/2653701099 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30b400700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c1046d0 0x7fb30c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 -- 172.21.15.26:0/2653701099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c1046d0 msgr2=0x7fb30c114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c1046d0 0x7fb30c114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 -- 172.21.15.26:0/2653701099 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8005040 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.565+0000 7fb30be00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc002a50 tx=0x7fb2fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.566+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2fc007cc0 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.566+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2fc00f040 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.566+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30c119660 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.566+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2fc00da30 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.566+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30c119c20 con 0x7fb30c103420 2024-04-16T07:22:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.567+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c04c710 con 0x7fb30c103420 2024-04-16T07:22:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.568+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb2fc00db90 con 0x7fb30c103420 2024-04-16T07:22:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.569+0000 7fb303e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 0x7fb2f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.569+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb2fc086f60 con 0x7fb30c103420 2024-04-16T07:22:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.569+0000 7fb30b400700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 0x7fb2f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.570+0000 7fb30b400700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 0x7fb2f006b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800b3b0 tx=0x7fb2f8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.574+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb2fc056c30 con 0x7fb30c103420 2024-04-16T07:22:03.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:03 smithi026 ceph-mon[28805]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:03.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:03 smithi026 ceph-mon[28805]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:03.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:03 smithi026 ceph-mon[28805]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:03 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3949576382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:22:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:03 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/4248443577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:22:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.813+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 --> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb30c108a10 con 0x7fb2f00692a0 2024-04-16T07:22:03.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.816+0000 7fb303e00700 1 -- 172.21.15.26:0/2653701099 <== mgr.14208 v2:172.21.15.26:6800/3943853548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fb30c108a10 con 0x7fb2f00692a0 2024-04-16T07:22:03.815 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:22:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:22:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "0/21 daemons upgraded", 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa image on host smithi104", 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:22:03.816 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.819+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 msgr2=0x7fb2f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.819+0000 7fb312e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 0x7fb2f006b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800b3b0 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.819+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 msgr2=0x7fb30c113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.819+0000 7fb312e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc002a50 tx=0x7fb2fc0046b0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 shutdown_connections 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7fb2f00692a0 0x7fb2f006b750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb30c103420 0x7fb30c113f50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 --2- 172.21.15.26:0/2653701099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb30c1046d0 0x7fb30c114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 >> 172.21.15.26:0/2653701099 conn(0x7fb30c072640 msgr2=0x7fb30c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 shutdown_connections 2024-04-16T07:22:03.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.820+0000 7fb312e00700 1 -- 172.21.15.26:0/2653701099 wait complete. 2024-04-16T07:22:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 -- 172.21.15.26:0/4164901724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 msgr2=0x7f7e440fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 --2- 172.21.15.26:0/4164901724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e440fd030 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7e38004940 tx=0x7f7e38009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 -- 172.21.15.26:0/4164901724 shutdown_connections 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 --2- 172.21.15.26:0/4164901724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e440fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 --2- 172.21.15.26:0/4164901724 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e440fd030 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 -- 172.21.15.26:0/4164901724 >> 172.21.15.26:0/4164901724 conn(0x7f7e440f83a0 msgr2=0x7f7e440fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 -- 172.21.15.26:0/4164901724 shutdown_connections 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.976+0000 7f7e4a400700 1 -- 172.21.15.26:0/4164901724 wait complete. 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.977+0000 7f7e4a400700 1 Processor -- start 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.977+0000 7f7e4a400700 1 -- start start 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e4a400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e4a400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e44070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e4a400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440708c0 con 0x7f7e440fcc20 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e4a400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e44070a30 con 0x7f7e440fded0 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47196/0 (socket says 172.21.15.26:47196) 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 -- 172.21.15.26:0/1588256404 learned_addr learned my addr 172.21.15.26:0/1588256404 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e43e00700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e44070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 -- 172.21.15.26:0/1588256404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 msgr2=0x7f7e44070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e44070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e49000700 1 -- 172.21.15.26:0/1588256404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38005040 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.978+0000 7f7e43e00700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e44070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e49000700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009150 tx=0x7f7e38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3800e040 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e3800bcc0 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e44070cb0 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3801c630 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.979+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e44115870 con 0x7f7e440fcc20 2024-04-16T07:22:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.980+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e440fc020 con 0x7f7e440fcc20 2024-04-16T07:22:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.981+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7e38013420 con 0x7f7e440fcc20 2024-04-16T07:22:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.982+0000 7f7e40c00700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 0x7f7e2c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.983+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7e38088090 con 0x7f7e440fcc20 2024-04-16T07:22:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.983+0000 7f7e43e00700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 0x7f7e2c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.984+0000 7f7e43e00700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 0x7f7e2c06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009fd0 tx=0x7f7e3400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:03.987+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7e380542d0 con 0x7f7e440fcc20 2024-04-16T07:22:03.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:03.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:22:03.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:22:03.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 7) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.996+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.997+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 11 ==== client_session(renewcaps seq 7) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:22:03.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:03.997+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 7) v1 from mds.0 2024-04-16T07:22:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:03 smithi104 ceph-mon[36163]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:03 smithi104 ceph-mon[36163]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:03 smithi104 ceph-mon[36163]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:03 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3949576382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:22:04.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:03 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/4248443577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:22:04.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:04.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:04.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:04.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:04.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:04.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:04.287+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:04.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.291+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7e4404bed0 con 0x7f7e440fcc20 2024-04-16T07:22:04.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.292+0000 7f7e40c00700 1 -- 172.21.15.26:0/1588256404 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f7e4404bed0 con 0x7f7e440fcc20 2024-04-16T07:22:04.291 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:22:04.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 msgr2=0x7f7e2c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:04.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 0x7f7e2c06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009fd0 tx=0x7f7e3400b040 comp rx=0 tx=0).stop 2024-04-16T07:22:04.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 msgr2=0x7f7e44071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:04.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009150 tx=0x7f7e38009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 shutdown_connections 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:6800/3943853548,v1:172.21.15.26:6801/3943853548] conn(0x7f7e2c0692a0 0x7f7e2c06b750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e440fcc20 0x7f7e44071e40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 --2- 172.21.15.26:0/1588256404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e440fded0 0x7f7e44070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.294+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 >> 172.21.15.26:0/1588256404 conn(0x7f7e440f83a0 msgr2=0x7f7e440f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.295+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 shutdown_connections 2024-04-16T07:22:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:04.295+0000 7f7e4a400700 1 -- 172.21.15.26:0/1588256404 wait complete. 2024-04-16T07:22:04.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:04.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:04.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:04.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:04.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:04.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:04.997+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:04 smithi026 ceph-mon[28805]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:04 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1588256404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:22:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:04 smithi104 ceph-mon[36163]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:04 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1588256404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:05.287+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:05.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:05.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:05.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:05.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:05.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:05.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:05.997+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:05 smithi026 ceph-mon[28805]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:05 smithi104 ceph-mon[36163]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:06.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:06.287+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:06.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:06.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:06.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:06.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:06.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:06.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:06.997+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:07.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:06 smithi026 ceph-mon[28805]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:06 smithi104 ceph-mon[36163]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:07.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:07.287+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:07.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:07.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:07.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:07.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:07.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:07.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:07.997+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:08.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:08.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:08.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:08.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:08.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:08.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:08.288+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:08.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:08.998+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:09.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:08 smithi026 ceph-mon[28805]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:09.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:08 smithi104 ceph-mon[36163]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:09.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:09.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:09.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:09.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:09.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:09.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:09.288+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:09.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:09.998+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:10.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: Upgrade: Updating mgr.smithi104.uqfljg 2024-04-16T07:22:10.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:10.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:22:10.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:22:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:22:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:09 smithi104 ceph-mon[36163]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:10.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: Upgrade: Updating mgr.smithi104.uqfljg 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:22:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:09 smithi026 ceph-mon[28805]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:10.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:10.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:10.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:10.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:10.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:10.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:10.288+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:10.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:10.998+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:11.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:11.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:11.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:11.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:11.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:11.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:11.288+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:11.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:11.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:11.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:11.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:11.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:11.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:11.999+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:12.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:12.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:22:12.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 7) v5 -- 0x7fe9ac0047d0 con 0x55fc3f51baa0 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.289+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 11 ==== client_session(renewcaps seq 7) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0047d0 con 0x55fc3f51baa0 2024-04-16T07:22:12.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:12.290+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 7) v1 from mds.0 2024-04-16T07:22:12.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:12 smithi104 ceph-mon[36163]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:12.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:12 smithi026 ceph-mon[28805]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:12.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:12.999+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:13.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:13.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:13.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:13.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:13.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:13.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:13.289+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:13.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:13.999+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:14.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:14.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:14.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:14.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:14.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:14.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:14.289+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:14 smithi026 ceph-mon[28805]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:14 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:14 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:14 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:14 smithi104 ceph-mon[36163]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:14 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:14 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:14 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:14.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:14.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:14.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:14.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:14.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:14.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:14.999+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:15.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:15.289+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:15 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:15.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:15 smithi104 ceph-mon[36163]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:15.999+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:15.999+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:15.999+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:15.999+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:16.000+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:15.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:16.000+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:16.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:16.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:15 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:16.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:15 smithi026 ceph-mon[28805]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:16.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:16.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:16.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:16.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:16.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:16.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:16.289+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:17.000+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:17.007 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:16 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:17.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:16 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:17.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.289+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:17.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:17.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.289+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:17.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.289+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:17.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.290+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:17.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:17.290+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:17.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:18.000+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:18.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:17 smithi026 ceph-mon[28805]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:18.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:17 smithi104 ceph-mon[36163]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:18.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:18.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:18.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:18.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:18.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:18.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:18.290+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:18.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:18.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:18.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:18.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:19.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:19.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:19.000+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:19.122 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:19.122 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:18 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:19.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:19.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:18 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:19.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:19.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:19.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:19.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:19.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:19.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:19.290+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:19.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.000+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:19.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.000+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.000+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.001+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.001+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:20.001+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:20.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:20.290+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:20 smithi026 ceph-mon[28805]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:20.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:20 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:20 smithi104 ceph-mon[36163]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:20 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:21.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:21.001+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:21.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:21.290+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:22.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:22.001+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:22.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.290+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:22.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:22.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.290+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:22.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.290+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 3 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:22.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.291+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:22.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:22.291+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:22 smithi026 ceph-mon[28805]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:22.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:22 smithi104 ceph-mon[36163]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:23.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:23.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:23.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:23.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:23.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:23.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:23.002+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:23.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 62835 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 12, success, outsize: 120 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.150+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.150+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.150+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.150+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.150+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 62918 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 14, success, outsize: 120 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.163+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.163+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.163+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.163+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.163+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63006 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.175+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.175+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.175+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 16, success, outsize: 120 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.175+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.175+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63108 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.188+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 18, success, outsize: 120 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.188+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.188+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.188+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.188+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63197 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 20, success, outsize: 120 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.200+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.200+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.200+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.200+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.200+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63280 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 22, success, outsize: 120 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.212+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.212+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.212+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.212+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.212+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63366 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 24, success, outsize: 120 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.224+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.224+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.224+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.224+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.224+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63458 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 26, success, outsize: 120 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.237+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.237+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.237+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.237+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.237+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63547 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 28, success, outsize: 120 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.250+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.250+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.250+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.250+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.250+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63630 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 30, success, outsize: 120 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.262+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.262+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.262+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.262+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.262+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63716 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 32, success, outsize: 120 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.275+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.275+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.275+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.275+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.275+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63808 2024-04-16T07:22:23.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 34, success, outsize: 120 2024-04-16T07:22:23.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.288+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.288+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.288+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.288+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.288+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 15 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:23.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.291+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63897 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 36, success, outsize: 120 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.301+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.301+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.301+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.301+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.301+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63980 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 38, success, outsize: 120 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.312+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.312+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.312+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.312+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.313+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64066 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 40, success, outsize: 120 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.325+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.325+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.325+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.325+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.325+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64158 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 42, success, outsize: 120 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.337+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.337+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.337+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.337+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.337+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64247 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 44, success, outsize: 120 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.350+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.350+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.350+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.350+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.350+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64330 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 46, success, outsize: 120 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.362+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.362+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.362+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.362+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.362+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64416 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 48, success, outsize: 120 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.375+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.375+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.375+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.375+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.375+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64508 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 50, success, outsize: 120 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.387+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.387+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.387+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.388+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.388+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64597 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 52, success, outsize: 120 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.400+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.400+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.400+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.400+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.400+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64680 2024-04-16T07:22:23.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 54, success, outsize: 120 2024-04-16T07:22:23.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.413+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.413+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.413+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.413+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.413+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64766 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 56, success, outsize: 120 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.427+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.427+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.427+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.427+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.427+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64858 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 58, success, outsize: 120 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.441+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.441+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.441+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.441+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.441+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64947 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 60, success, outsize: 120 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.455+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.455+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.455+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.455+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.456+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65030 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 62, success, outsize: 120 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.468+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.469+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.469+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.469+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.469+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65116 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 64, success, outsize: 120 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.483+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.483+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.483+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.483+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.483+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65208 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.499+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 66, success, outsize: 120 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.499+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.499+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.499+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:23.500+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:23.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:22:23.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:22:23.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:22:23.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:22:23.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:22:23.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:23 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:22:23.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:22:23.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:22:23.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:22:23.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:22:23.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:22:23.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:23 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 8) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:24.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:24.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 12 ==== client_session(renewcaps seq 8) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:22:24.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:24.002+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 8) v1 from mds.0 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:24.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:24.291+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:24.662 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:24 smithi026 ceph-mon[28805]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:24.662 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:24 smithi026 ceph-mon[28805]: mgrmap e20: smithi026.tawexg(active, since 8m), standbys: smithi104.uqfljg 2024-04-16T07:22:24.662 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:24 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:24 smithi104 ceph-mon[36163]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:24 smithi104 ceph-mon[36163]: mgrmap e20: smithi026.tawexg(active, since 8m), standbys: smithi104.uqfljg 2024-04-16T07:22:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:24 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:25.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:25.002+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:25.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:25.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:25.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:25.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:25.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:25.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:25.291+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr fail", "who": "smithi026.tawexg"}]: dispatch 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: Activating manager daemon smithi104.uqfljg 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "mgr fail", "who": "smithi026.tawexg"}]': finished 2024-04-16T07:22:25.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: mgrmap e21: smithi104.uqfljg(active, starting, since 0.0151617s) 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:22:25.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: Manager daemon smithi104.uqfljg is now available 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.uqfljg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:22:25.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.uqfljg/trash_purge_schedule"}]: dispatch 2024-04-16T07:22:25.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr fail", "who": "smithi026.tawexg"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: Activating manager daemon smithi104.uqfljg 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.14208 172.21.15.26:0/907003072' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "mgr fail", "who": "smithi026.tawexg"}]': finished 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: mgrmap e21: smithi104.uqfljg(active, starting, since 0.0151617s) 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:22:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: Manager daemon smithi104.uqfljg is now available 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.uqfljg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:22:25.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.uqfljg/trash_purge_schedule"}]: dispatch 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:26.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:26.002+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:26.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:26.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:26.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:26.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:26.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:26.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:26.291+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:26 smithi026 ceph-mon[28805]: Deploying cephadm binary to smithi104 2024-04-16T07:22:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:26 smithi026 ceph-mon[28805]: Deploying cephadm binary to smithi026 2024-04-16T07:22:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:26 smithi026 ceph-mon[28805]: mgrmap e22: smithi104.uqfljg(active, since 1.01591s) 2024-04-16T07:22:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:26 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:26 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:26 smithi104 ceph-mon[36163]: Deploying cephadm binary to smithi104 2024-04-16T07:22:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:26 smithi104 ceph-mon[36163]: Deploying cephadm binary to smithi026 2024-04-16T07:22:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:26 smithi104 ceph-mon[36163]: mgrmap e22: smithi104.uqfljg(active, since 1.01591s) 2024-04-16T07:22:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:26 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:26 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:27.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:27.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:27.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:27.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:27.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:27.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:27.002+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:27.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:27.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:27.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:27.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:27.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:27.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:27.292+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:27 smithi026 ceph-mon[28805]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:27.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:27 smithi104 ceph-mon[36163]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:28.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:28.003+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:28.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:28.292+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:28 smithi104 ceph-mon[36163]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:28 smithi104 ceph-mon[36163]: [16/Apr/2024:07:22:27] ENGINE Bus STARTING 2024-04-16T07:22:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:28 smithi104 ceph-mon[36163]: [16/Apr/2024:07:22:27] ENGINE Serving on http://172.21.15.104:8765 2024-04-16T07:22:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:28 smithi104 ceph-mon[36163]: [16/Apr/2024:07:22:27] ENGINE Serving on https://172.21.15.104:7150 2024-04-16T07:22:28.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:28 smithi104 ceph-mon[36163]: [16/Apr/2024:07:22:27] ENGINE Bus STARTED 2024-04-16T07:22:29.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:29.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:29.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:29.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:29.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:29.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:29.003+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:28 smithi026 ceph-mon[28805]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:28 smithi026 ceph-mon[28805]: [16/Apr/2024:07:22:27] ENGINE Bus STARTING 2024-04-16T07:22:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:28 smithi026 ceph-mon[28805]: [16/Apr/2024:07:22:27] ENGINE Serving on http://172.21.15.104:8765 2024-04-16T07:22:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:28 smithi026 ceph-mon[28805]: [16/Apr/2024:07:22:27] ENGINE Serving on https://172.21.15.104:7150 2024-04-16T07:22:29.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:28 smithi026 ceph-mon[28805]: [16/Apr/2024:07:22:27] ENGINE Bus STARTED 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:29.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:29.292+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:30.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:30.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:30.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:30.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:30.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:30.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:30.003+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:30.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:30.292+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:30 smithi104 ceph-mon[36163]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:30 smithi104 ceph-mon[36163]: mgrmap e23: smithi104.uqfljg(active, since 4s) 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:31.003+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:30 smithi026 ceph-mon[28805]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:30 smithi026 ceph-mon[28805]: mgrmap e23: smithi104.uqfljg(active, since 4s) 2024-04-16T07:22:31.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:31.293+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.003+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.004+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.004+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.004+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.004+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:32.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:32.004+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 8) v5 -- 0x7fe9ac0047d0 con 0x55fc3f51baa0 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:32.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:32.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 12 ==== client_session(renewcaps seq 8) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:32.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:32.294+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 8) v1 from mds.0 2024-04-16T07:22:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:32 smithi104 ceph-mon[36163]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T07:22:32.900 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:32 smithi026 ceph-mon[28805]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:33.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:33.004+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:33.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:33.293+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:34.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:34.004+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:34.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:34.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:34.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:34.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:34.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:34.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:34.294+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:34.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.462+0000 7fd563200700 1 -- 172.21.15.26:0/2113716143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c071f50 msgr2=0x7fd55c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.462+0000 7fd563200700 1 --2- 172.21.15.26:0/2113716143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c071f50 0x7fd55c070460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd54c004940 tx=0x7fd54c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 -- 172.21.15.26:0/2113716143 shutdown_connections 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 --2- 172.21.15.26:0/2113716143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0709a0 0x7fd55c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 --2- 172.21.15.26:0/2113716143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c071f50 0x7fd55c070460 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 -- 172.21.15.26:0/2113716143 >> 172.21.15.26:0/2113716143 conn(0x7fd55c0f8530 msgr2=0x7fd55c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 -- 172.21.15.26:0/2113716143 shutdown_connections 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.463+0000 7fd563200700 1 -- 172.21.15.26:0/2113716143 wait complete. 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 Processor -- start 2024-04-16T07:22:34.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 -- start start 2024-04-16T07:22:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c071f50 0x7fd55c110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c110870 con 0x7fd55c0709a0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.464+0000 7fd563200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c110fb0 con 0x7fd55c071f50 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46188/0 (socket says 172.21.15.26:46188) 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 -- 172.21.15.26:0/4232949424 learned_addr learned my addr 172.21.15.26:0/4232949424 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd553e00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c071f50 0x7fd55c110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 -- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c071f50 msgr2=0x7fd55c110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c071f50 0x7fd55c110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.465+0000 7fd55be00700 1 -- 172.21.15.26:0/4232949424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c005040 con 0x7fd55c0709a0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd55be00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b970 tx=0x7fd54c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54c00e040 con 0x7fd55c0709a0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd54c00bbf0 con 0x7fd55c0709a0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54c01c630 con 0x7fd55c0709a0 2024-04-16T07:22:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c111230 con 0x7fd55c0709a0 2024-04-16T07:22:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.466+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c1117f0 con 0x7fd55c0709a0 2024-04-16T07:22:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.467+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) v1 ==== 46568+0+0 (secure 0 0 0) 0x7fd54c00bd60 con 0x7fd55c0709a0 2024-04-16T07:22:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.467+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c05f5f0 con 0x7fd55c0709a0 2024-04-16T07:22:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.468+0000 7fd558c00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 0x7fd54003b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.468+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fd54c04e670 con 0x7fd55c0709a0 2024-04-16T07:22:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.468+0000 7fd553e00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 0x7fd54003b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.469+0000 7fd553e00700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 0x7fd54003b930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd544007fd0 tx=0x7fd544009040 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.474+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fd54c011940 con 0x7fd55c0709a0 2024-04-16T07:22:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.557+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fd54c02bcf0 con 0x7fd55c0709a0 2024-04-16T07:22:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.739+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd55c111b70 con 0x7fd540039480 2024-04-16T07:22:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.743+0000 7fd558c00700 1 -- 172.21.15.26:0/4232949424 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fd55c111b70 con 0x7fd540039480 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 msgr2=0x7fd54003b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 0x7fd54003b930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd544007fd0 tx=0x7fd544009040 comp rx=0 tx=0).stop 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 msgr2=0x7fd55c10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b970 tx=0x7fd54c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 shutdown_connections 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fd540039480 0x7fd54003b930 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd55c0709a0 0x7fd55c10fd30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 --2- 172.21.15.26:0/4232949424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c071f50 0x7fd55c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 >> 172.21.15.26:0/4232949424 conn(0x7fd55c0f8530 msgr2=0x7fd55c0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 shutdown_connections 2024-04-16T07:22:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.746+0000 7fd563200700 1 -- 172.21.15.26:0/4232949424 wait complete. 2024-04-16T07:22:34.759 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: Standby manager daemon smithi026.tawexg started 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/crt"}]: dispatch 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/key"}]: dispatch 2024-04-16T07:22:34.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:34 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:22:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.898+0000 7f4fea400700 1 -- 172.21.15.26:0/3383370537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 msgr2=0x7f4fe40ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.898+0000 7f4fea400700 1 --2- 172.21.15.26:0/3383370537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe40ff420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4004940 tx=0x7f4fd4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: Standby manager daemon smithi026.tawexg started 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/crt"}]: dispatch 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/key"}]: dispatch 2024-04-16T07:22:34.898 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:34 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/1577502207' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.899+0000 7f4fea400700 1 -- 172.21.15.26:0/3383370537 shutdown_connections 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.899+0000 7f4fea400700 1 --2- 172.21.15.26:0/3383370537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.899+0000 7f4fea400700 1 --2- 172.21.15.26:0/3383370537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe40ff420 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.899+0000 7f4fea400700 1 -- 172.21.15.26:0/3383370537 >> 172.21.15.26:0/3383370537 conn(0x7f4fe40fa790 msgr2=0x7f4fe40fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.899+0000 7f4fea400700 1 -- 172.21.15.26:0/3383370537 shutdown_connections 2024-04-16T07:22:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.900+0000 7f4fea400700 1 -- 172.21.15.26:0/3383370537 wait complete. 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.900+0000 7f4fea400700 1 Processor -- start 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.901+0000 7f4fea400700 1 -- start start 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.901+0000 7f4fea400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe410fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.901+0000 7f4fea400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.901+0000 7f4fea400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4110670 con 0x7f4fe40ff010 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.901+0000 7f4fea400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe41107e0 con 0x7f4fe41002c0 2024-04-16T07:22:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe410fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.26:37284/0 (socket says 172.21.15.26:37284) 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 -- 172.21.15.26:0/1342998555 learned_addr learned my addr 172.21.15.26:0/1342998555 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 -- 172.21.15.26:0/1342998555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 msgr2=0x7f4fe410fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe410fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.902+0000 7f4fe2a00700 1 -- 172.21.15.26:0/1342998555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4005040 con 0x7f4fe41002c0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.903+0000 7f4fe2a00700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c830 tx=0x7f4fd0004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.903+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd00106f0 con 0x7f4fe41002c0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.903+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe4115240 con 0x7f4fe41002c0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.904+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fd0010d20 con 0x7f4fe41002c0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.904+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd001bea0 con 0x7f4fe41002c0 2024-04-16T07:22:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.904+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe4115800 con 0x7f4fe41002c0 2024-04-16T07:22:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.905+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe404c710 con 0x7f4fe41002c0 2024-04-16T07:22:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.907+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f4fd001b400 con 0x7f4fe41002c0 2024-04-16T07:22:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.907+0000 7f4fdb400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 0x7f4fc806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.908+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f4fd008b420 con 0x7f4fe41002c0 2024-04-16T07:22:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.908+0000 7f4fe3400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 0x7f4fc806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.910+0000 7f4fe3400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 0x7f4fc806e1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400b3b0 tx=0x7f4fd4009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:34.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:34.924+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f4fd0058620 con 0x7f4fe41002c0 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:35.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:35.004+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:35.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.186+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fe4104630 con 0x7f4fc806bd00 2024-04-16T07:22:35.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.189+0000 7f4fdb400700 1 -- 172.21.15.26:0/1342998555 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f4fe4104630 con 0x7f4fc806bd00 2024-04-16T07:22:35.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.191+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 msgr2=0x7f4fc806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 0x7f4fc806e1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400b3b0 tx=0x7f4fd4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 msgr2=0x7f4fe4110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c830 tx=0x7f4fd0004470 comp rx=0 tx=0).stop 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 shutdown_connections 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4fc806bd00 0x7f4fc806e1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fe40ff010 0x7f4fe410fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 --2- 172.21.15.26:0/1342998555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fe41002c0 0x7f4fe4110070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 >> 172.21.15.26:0/1342998555 conn(0x7f4fe40fa790 msgr2=0x7f4fe40fbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 shutdown_connections 2024-04-16T07:22:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.192+0000 7f4fea400700 1 -- 172.21.15.26:0/1342998555 wait complete. 2024-04-16T07:22:35.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:35.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:35.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:35.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:35.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:35.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:35.294+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 -- 172.21.15.26:0/1445282848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 msgr2=0x7f4dc8070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/1445282848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc8070d60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4db4004940 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 -- 172.21.15.26:0/1445282848 shutdown_connections 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/1445282848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc8070d60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/1445282848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 0x7f4dc80703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 -- 172.21.15.26:0/1445282848 >> 172.21.15.26:0/1445282848 conn(0x7f4dc80f8530 msgr2=0x7f4dc80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 -- 172.21.15.26:0/1445282848 shutdown_connections 2024-04-16T07:22:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.342+0000 7f4dcfa00700 1 -- 172.21.15.26:0/1445282848 wait complete. 2024-04-16T07:22:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.343+0000 7f4dcfa00700 1 Processor -- start 2024-04-16T07:22:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.343+0000 7f4dcfa00700 1 -- start start 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcfa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcfa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 0x7f4dc8110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcfa00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc8110760 con 0x7f4dc80708f0 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcfa00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc8110fb0 con 0x7f4dc8071ea0 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcd000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcd000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46236/0 (socket says 172.21.15.26:46236) 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.344+0000 7f4dcd000700 1 -- 172.21.15.26:0/3204825795 learned_addr learned my addr 172.21.15.26:0/3204825795 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dc7e00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 0x7f4dc8110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dcd000700 1 -- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 msgr2=0x7f4dc8110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dcd000700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 0x7f4dc8110160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dcd000700 1 -- 172.21.15.26:0/3204825795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4005040 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dcd000700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c8b0 tx=0x7f4dbc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dbc00a760 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dbc00ad90 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dbc0113a0 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.345+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc8111290 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.346+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc81117d0 con 0x7f4dc80708f0 2024-04-16T07:22:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.347+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc805f5f0 con 0x7f4dc80708f0 2024-04-16T07:22:35.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.348+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f4dbc012400 con 0x7f4dc80708f0 2024-04-16T07:22:35.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.348+0000 7f4dc4c00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 0x7f4db006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.349+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f4dbc08a9a0 con 0x7f4dc80708f0 2024-04-16T07:22:35.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.349+0000 7f4dc7e00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 0x7f4db006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.350+0000 7f4dc7e00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 0x7f4db006e1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4db400b3b0 tx=0x7f4db4009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.353+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f4dbc057ba0 con 0x7f4dc80708f0 2024-04-16T07:22:35.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.607+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc8111c10 con 0x7f4db006bcf0 2024-04-16T07:22:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.639+0000 7f4dc4c00700 1 -- 172.21.15.26:0/3204825795 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f4dc8111c10 con 0x7f4db006bcf0 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (8m) 2m ago 9m 13.1M - 0.25.0 c8568f914cd2 d5f7c216b68d 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (10m) 2m ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (8m) 17s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (7m) 2m ago 9m 73.0M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (2m) 2m ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (2m) 2m ago 2m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (2m) 17s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (2m) 17s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (11m) 2m ago 11m 481M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (22s) 17s ago 8m 58.3M - 19.0.0-2884-g218a6914 a4e4811815e4 c49f0b0c525f 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (11m) 2m ago 11m 44.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:22:35.639 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (8m) 17s ago 8m 31.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (10m) 2m ago 10m 8074k - 1.5.0 0da6a335fe13 5198001b496e 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 17s ago 8m 7620k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (7m) 2m ago 7m 43.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (6m) 2m ago 6m 46.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (6m) 2m ago 6m 44.1M 4290M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (5m) 17s ago 5m 44.0M 5312M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (5m) 17s ago 5m 42.4M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (4m) 17s ago 4m 41.7M 5312M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:22:35.640 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (7m) 2m ago 9m 46.4M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 msgr2=0x7f4db006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 0x7f4db006e1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4db400b3b0 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 msgr2=0x7f4dc810fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c8b0 tx=0x7f4dbc004470 comp rx=0 tx=0).stop 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 shutdown_connections 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f4db006bcf0 0x7f4db006e1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4dc80708f0 0x7f4dc810fc20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 --2- 172.21.15.26:0/3204825795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc8071ea0 0x7f4dc8110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 >> 172.21.15.26:0/3204825795 conn(0x7f4dc80f8530 msgr2=0x7f4dc80f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 shutdown_connections 2024-04-16T07:22:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.642+0000 7f4dcfa00700 1 -- 172.21.15.26:0/3204825795 wait complete. 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.788+0000 7fb557e00700 1 -- 172.21.15.26:0/2295966515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fcc30 msgr2=0x7fb5580fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.788+0000 7fb557e00700 1 --2- 172.21.15.26:0/2295966515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fcc30 0x7fb5580fd040 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb548004940 tx=0x7fb548009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 -- 172.21.15.26:0/2295966515 shutdown_connections 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 --2- 172.21.15.26:0/2295966515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fdee0 0x7fb5580fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 --2- 172.21.15.26:0/2295966515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fcc30 0x7fb5580fd040 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 -- 172.21.15.26:0/2295966515 >> 172.21.15.26:0/2295966515 conn(0x7fb5580f83d0 msgr2=0x7fb5580fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 -- 172.21.15.26:0/2295966515 shutdown_connections 2024-04-16T07:22:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.789+0000 7fb557e00700 1 -- 172.21.15.26:0/2295966515 wait complete. 2024-04-16T07:22:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 Processor -- start 2024-04-16T07:22:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 -- start start 2024-04-16T07:22:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fcc30 0x7fb55810d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55810e440 con 0x7fb5580fdee0 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.790+0000 7fb557e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55810ec30 con 0x7fb5580fcc30 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46266/0 (socket says 172.21.15.26:46266) 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 -- 172.21.15.26:0/1727375603 learned_addr learned my addr 172.21.15.26:0/1727375603 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556a00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fcc30 0x7fb55810d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 -- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fcc30 msgr2=0x7fb55810d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fcc30 0x7fb55810d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.791+0000 7fb556000700 1 -- 172.21.15.26:0/1727375603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548005040 con 0x7fb5580fdee0 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.792+0000 7fb556000700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb54400b790 tx=0x7fb544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.792+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54400f770 con 0x7fb5580fdee0 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.792+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb54400fda0 con 0x7fb5580fdee0 2024-04-16T07:22:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.792+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb54400d310 con 0x7fb5580fdee0 2024-04-16T07:22:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.793+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55810ef10 con 0x7fb5580fdee0 2024-04-16T07:22:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.793+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55810f4a0 con 0x7fb5580fdee0 2024-04-16T07:22:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.795+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55805f5f0 con 0x7fb5580fdee0 2024-04-16T07:22:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.795+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fb544010420 con 0x7fb5580fdee0 2024-04-16T07:22:35.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.796+0000 7fb54ea00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 0x7fb53c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:35.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.796+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb544089ed0 con 0x7fb5580fdee0 2024-04-16T07:22:35.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.797+0000 7fb556a00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 0x7fb53c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.798+0000 7fb556a00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 0x7fb53c06e1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb548004f90 tx=0x7fb548009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:35.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:35.801+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fb54404c930 con 0x7fb5580fdee0 2024-04-16T07:22:35.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:35 smithi104 ceph-mon[36163]: mgrmap e24: smithi104.uqfljg(active, since 9s), standbys: smithi026.tawexg 2024-04-16T07:22:35.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:35 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:22:35.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:35 smithi104 ceph-mon[36163]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:35.946 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:35 smithi026 ceph-mon[28805]: mgrmap e24: smithi104.uqfljg(active, since 9s), standbys: smithi026.tawexg 2024-04-16T07:22:35.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:35 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:22:35.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:35 smithi026 ceph-mon[28805]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:36.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:36.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:36.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:36.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:36.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:36.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:36.004+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.136+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb55810f770 con 0x7fb5580fdee0 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.138+0000 7fb54ea00700 1 -- 172.21.15.26:0/1727375603 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7fb5440570d0 con 0x7fb5580fdee0 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:36.137 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 13, 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:22:36.138 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:22:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.140+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 msgr2=0x7fb53c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.140+0000 7fb557e00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 0x7fb53c06e1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb548004f90 tx=0x7fb548009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.140+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 msgr2=0x7fb55810ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.140+0000 7fb557e00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb54400b790 tx=0x7fb544009250 comp rx=0 tx=0).stop 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 shutdown_connections 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb53c06bcf0 0x7fb53c06e1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580fcc30 0x7fb55810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 --2- 172.21.15.26:0/1727375603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5580fdee0 0x7fb55810ddb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 >> 172.21.15.26:0/1727375603 conn(0x7fb5580f83d0 msgr2=0x7fb5580f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 shutdown_connections 2024-04-16T07:22:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.141+0000 7fb557e00700 1 -- 172.21.15.26:0/1727375603 wait complete. 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:36.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:36.294+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:36.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.300+0000 7ff2e2200700 1 -- 172.21.15.26:0/2310241161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0662f0 msgr2=0x7ff2dc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.300+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2310241161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0ffa30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc004940 tx=0x7ff2cc009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 -- 172.21.15.26:0/2310241161 shutdown_connections 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2310241161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2310241161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0ffa30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 -- 172.21.15.26:0/2310241161 >> 172.21.15.26:0/2310241161 conn(0x7ff2dc0f74c0 msgr2=0x7ff2dc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 -- 172.21.15.26:0/2310241161 shutdown_connections 2024-04-16T07:22:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.301+0000 7ff2e2200700 1 -- 172.21.15.26:0/2310241161 wait complete. 2024-04-16T07:22:36.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.302+0000 7ff2e2200700 1 Processor -- start 2024-04-16T07:22:36.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.303+0000 7ff2e2200700 1 -- start start 2024-04-16T07:22:36.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.303+0000 7ff2e2200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.303+0000 7ff2e2200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.303+0000 7ff2e2200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0fbee0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.303+0000 7ff2e2200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0fc050 con 0x7ff2dc0662f0 2024-04-16T07:22:36.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46282/0 (socket says 172.21.15.26:46282) 2024-04-16T07:22:36.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 -- 172.21.15.26:0/2384445371 learned_addr learned my addr 172.21.15.26:0/2384445371 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:36.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2db400700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 -- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0662f0 msgr2=0x7ff2dc0fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0fd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 -- 172.21.15.26:0/2384445371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc005040 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.304+0000 7ff2daa00700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8002a50 tx=0x7ff2c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.305+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2c8007cc0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.305+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2c800f040 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.305+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2c8009640 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.305+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc0fc330 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.305+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc115a90 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.306+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc05f5f0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.307+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7ff2c80097a0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.308+0000 7ff2d3400700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 0x7ff2c006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.308+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ff2c808a880 con 0x7ff2dc0fff70 2024-04-16T07:22:36.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.309+0000 7ff2db400700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 0x7ff2c006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.310+0000 7ff2db400700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 0x7ff2c006e1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc004890 tx=0x7ff2cc009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:36.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.313+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7ff2c8053dc0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.625+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff2dc1058b0 con 0x7ff2dc0fff70 2024-04-16T07:22:36.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.625+0000 7ff2d3400700 1 -- 172.21.15.26:0/2384445371 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7ff2c8057480 con 0x7ff2dc0fff70 2024-04-16T07:22:36.624 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:22:36.624 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:22:36.624 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:22:36.625 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:36.626 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 msgr2=0x7ff2c006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 0x7ff2c006e1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc004890 tx=0x7ff2cc009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 msgr2=0x7ff2dc0fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8002a50 tx=0x7ff2c80046b0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 shutdown_connections 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7ff2c006bcf0 0x7ff2c006e1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc0662f0 0x7ff2dc0fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 --2- 172.21.15.26:0/2384445371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2dc0fff70 0x7ff2dc0fb930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 >> 172.21.15.26:0/2384445371 conn(0x7ff2dc0f74c0 msgr2=0x7ff2dc0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 shutdown_connections 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.628+0000 7ff2e2200700 1 -- 172.21.15.26:0/2384445371 wait complete. 2024-04-16T07:22:36.628 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:22:36.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.780+0000 7f562f400700 1 -- 172.21.15.26:0/1749995922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 msgr2=0x7f5628070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.780+0000 7f562f400700 1 --2- 172.21.15.26:0/1749995922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628070460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f561c004940 tx=0x7f561c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 -- 172.21.15.26:0/1749995922 shutdown_connections 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 --2- 172.21.15.26:0/1749995922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 0x7f5628070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 --2- 172.21.15.26:0/1749995922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628070460 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 -- 172.21.15.26:0/1749995922 >> 172.21.15.26:0/1749995922 conn(0x7f56280f8530 msgr2=0x7f56280fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 -- 172.21.15.26:0/1749995922 shutdown_connections 2024-04-16T07:22:36.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.781+0000 7f562f400700 1 -- 172.21.15.26:0/1749995922 wait complete. 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 Processor -- start 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 -- start start 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 0x7f562810fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628110870 con 0x7f5628071f50 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f562f400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628110fb0 con 0x7f56280709a0 2024-04-16T07:22:36.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.782+0000 7f5627e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46302/0 (socket says 172.21.15.26:46302) 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 -- 172.21.15.26:0/1417199738 learned_addr learned my addr 172.21.15.26:0/1417199738 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f562ca00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 0x7f562810fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 -- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 msgr2=0x7f562810fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 0x7f562810fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 -- 172.21.15.26:0/1417199738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c005040 con 0x7f5628071f50 2024-04-16T07:22:36.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.783+0000 7f5627e00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5614002a50 tx=0x7f56140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:36.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.784+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5614007cc0 con 0x7f5628071f50 2024-04-16T07:22:36.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.784+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f561400f040 con 0x7f5628071f50 2024-04-16T07:22:36.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.784+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f561400da30 con 0x7f5628071f50 2024-04-16T07:22:36.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.784+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5628111290 con 0x7f5628071f50 2024-04-16T07:22:36.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.784+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5628111850 con 0x7f5628071f50 2024-04-16T07:22:36.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.785+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562805f5f0 con 0x7f5628071f50 2024-04-16T07:22:36.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.786+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f561400db90 con 0x7f5628071f50 2024-04-16T07:22:36.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.787+0000 7f5624c00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 0x7f561006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:36.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.787+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f561408a910 con 0x7f5628071f50 2024-04-16T07:22:36.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.788+0000 7f562ca00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 0x7f561006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:36.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.789+0000 7f562ca00700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 0x7f561006e1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b970 tx=0x7f561c009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:36.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:36.792+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f5614057900 con 0x7f5628071f50 2024-04-16T07:22:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:36 smithi104 ceph-mon[36163]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:22:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:36 smithi104 ceph-mon[36163]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:36 smithi104 ceph-mon[36163]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:36 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/1727375603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:22:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:37.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:37.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:37.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:37.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:37.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:37.005+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:36 smithi026 ceph-mon[28805]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:22:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:36 smithi026 ceph-mon[28805]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:36 smithi026 ceph-mon[28805]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:36 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/1727375603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:22:37.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.044+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5628111b80 con 0x7f561006bcf0 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.047+0000 7f5624c00700 1 -- 172.21.15.26:0/1417199738 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f5628111b80 con 0x7f561006bcf0 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:22:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T07:22:37.047 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "", 2024-04-16T07:22:37.047 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:22:37.047 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 msgr2=0x7f561006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 0x7f561006e1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b970 tx=0x7f561c009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 msgr2=0x7f5628110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5614002a50 tx=0x7f56140046b0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 shutdown_connections 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f561006bcf0 0x7f561006e1a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56280709a0 0x7f562810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 --2- 172.21.15.26:0/1417199738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5628071f50 0x7f5628110270 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 >> 172.21.15.26:0/1417199738 conn(0x7f56280f8530 msgr2=0x7f56280f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 shutdown_connections 2024-04-16T07:22:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.050+0000 7f562f400700 1 -- 172.21.15.26:0/1417199738 wait complete. 2024-04-16T07:22:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 -- 172.21.15.26:0/3997996167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 msgr2=0x7f58b0101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3997996167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b0101e30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f58a4004940 tx=0x7f58a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 -- 172.21.15.26:0/3997996167 shutdown_connections 2024-04-16T07:22:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3997996167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58b0102370 0x7f58b0104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3997996167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b0101e30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 -- 172.21.15.26:0/3997996167 >> 172.21.15.26:0/3997996167 conn(0x7f58b00f74c0 msgr2=0x7f58b00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:37.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 -- 172.21.15.26:0/3997996167 shutdown_connections 2024-04-16T07:22:37.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.201+0000 7f58b7e00700 1 -- 172.21.15.26:0/3997996167 wait complete. 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.202+0000 7f58b7e00700 1 Processor -- start 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.202+0000 7f58b7e00700 1 -- start start 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b7e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b7e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58b0102370 0x7f58b01101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b7e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b0110860 con 0x7f58b00662f0 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b7e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b0111000 con 0x7f58b0102370 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46330/0 (socket says 172.21.15.26:46330) 2024-04-16T07:22:37.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 -- 172.21.15.26:0/3897454436 learned_addr learned my addr 172.21.15.26:0/3897454436 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:22:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 -- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58b0102370 msgr2=0x7f58b01101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58b0102370 0x7f58b01101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.203+0000 7f58b5400700 1 -- 172.21.15.26:0/3897454436 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4005040 con 0x7f58b00662f0 2024-04-16T07:22:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58b5400700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f58a400b970 tx=0x7f58a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a400e040 con 0x7f58b00662f0 2024-04-16T07:22:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a400bbf0 con 0x7f58b00662f0 2024-04-16T07:22:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b0111280 con 0x7f58b00662f0 2024-04-16T07:22:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a401c630 con 0x7f58b00662f0 2024-04-16T07:22:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.204+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b0111840 con 0x7f58b00662f0 2024-04-16T07:22:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.205+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b005f5f0 con 0x7f58b00662f0 2024-04-16T07:22:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.206+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f58a400bd60 con 0x7f58b00662f0 2024-04-16T07:22:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.207+0000 7f58ad600700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 0x7f589806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:22:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.207+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f58a408bf20 con 0x7f58b00662f0 2024-04-16T07:22:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.208+0000 7f58b4a00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 0x7f589806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:22:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.209+0000 7f58b4a00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 0x7f589806e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c006db0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:22:37.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.212+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f58a4059040 con 0x7f58b00662f0 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:37.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:37.294+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:37.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.531+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f58b0107b30 con 0x7f58b00662f0 2024-04-16T07:22:37.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.532+0000 7f58ad600700 1 -- 172.21.15.26:0/3897454436 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f58a4058bd0 con 0x7f58b00662f0 2024-04-16T07:22:37.531 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 msgr2=0x7f589806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 0x7f589806e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c006db0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 msgr2=0x7f58b010fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f58a400b970 tx=0x7f58a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 shutdown_connections 2024-04-16T07:22:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f589806bcf0 0x7f589806e1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b00662f0 0x7f58b010fc90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 --2- 172.21.15.26:0/3897454436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58b0102370 0x7f58b01101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:22:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.534+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 >> 172.21.15.26:0/3897454436 conn(0x7f58b00f74c0 msgr2=0x7f58b00f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:22:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.535+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 shutdown_connections 2024-04-16T07:22:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:22:37.535+0000 7f58b7e00700 1 -- 172.21.15.26:0/3897454436 wait complete. 2024-04-16T07:22:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:37 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2384445371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:22:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:37 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:37 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:37 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3897454436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:38.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:38.005+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:38.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:37 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2384445371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:22:38.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:37 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:38.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:37 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:38.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:37 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3897454436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:22:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:38.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:38.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:38.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:38.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:38.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:38.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:38 smithi104 ceph-mon[36163]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:22:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:38 smithi104 ceph-mon[36163]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:39.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:39.005+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:38 smithi026 ceph-mon[28805]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:22:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:38 smithi026 ceph-mon[28805]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:22:39.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:39.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:39.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:39.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:39.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:39.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:39.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:40.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:40.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:40.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:40.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:40.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:40.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:40.006+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:40.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi026 to 1559M 2024-04-16T07:22:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:40.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:40.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:22:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi026 to 1559M 2024-04-16T07:22:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:41.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:41.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:41.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:41.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:41.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:41.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:41.006+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:41.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:40 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:41.274 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:40 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:41.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.294+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:41.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:41.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:41.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:41.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:41.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:41.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:42.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:42.007+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:42.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:41 smithi026 ceph-mon[28805]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:22:42.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:42.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:42.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:42.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:42.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:42.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:42.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:42.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:41 smithi104 ceph-mon[36163]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:22:43.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:43.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:43.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:43.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:43.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:43.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:43.006+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:43.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:43.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:43.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:43 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:43.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:43 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:43.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:43 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:43.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:43 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:44.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:44.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 9) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:44.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:44.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 13 ==== client_session(renewcaps seq 9) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:22:44.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:44.007+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 9) v1 from mds.0 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:44.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:44.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:44 smithi026 ceph-mon[28805]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:44 smithi104 ceph-mon[36163]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:45.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:45.007+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:45.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:45.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:45.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:45.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:45.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:45.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:45.295+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:45.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:45 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:45.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:45 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:46.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:46.007+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:46.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:45 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:46.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:45 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:46.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:46.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:46.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:46.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:46.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:46.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:46.297+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:46.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:46 smithi104 ceph-mon[36163]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:47.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:47.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:47.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:47.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:47.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:47.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:47.007+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:47.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:46 smithi026 ceph-mon[28805]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:47.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:47.296+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:48.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:48.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:48.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:48.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:48.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:48.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:48.008+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:48.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.296+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:48.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66570 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 68, success, outsize: 120 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.465+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.465+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.465+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.465+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.465+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66659 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.482+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 70, success, outsize: 120 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.482+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.482+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.482+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.482+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66751 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 72, success, outsize: 120 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.501+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.501+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.501+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.501+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.501+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66849 2024-04-16T07:22:48.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 74, success, outsize: 120 2024-04-16T07:22:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.520+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.520+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.520+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.520+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.520+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66944 2024-04-16T07:22:48.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 76, success, outsize: 120 2024-04-16T07:22:48.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.537+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.537+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.537+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.537+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.537+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67033 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 78, success, outsize: 120 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.554+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.554+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.554+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.554+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.554+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67125 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 80, success, outsize: 120 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.574+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.574+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.574+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.574+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.574+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67223 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 82, success, outsize: 120 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.589+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.589+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.589+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.589+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.589+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67318 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 84, success, outsize: 120 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.605+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.605+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.605+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.605+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.605+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67407 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 86, success, outsize: 120 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.620+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.620+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.620+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.620+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.620+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67499 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 88, success, outsize: 120 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.635+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.636+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.636+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.636+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.636+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67597 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 90, success, outsize: 120 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.651+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.651+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.651+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.651+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.651+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67692 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 92, success, outsize: 120 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.667+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.667+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.667+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.667+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.667+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67781 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 94, success, outsize: 120 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.682+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.682+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.682+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.682+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.682+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67873 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 96, success, outsize: 120 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.698+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.698+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.698+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.698+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.698+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.714 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67971 2024-04-16T07:22:48.714 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 98, success, outsize: 120 2024-04-16T07:22:48.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.715+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.715+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.715+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.715+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.715+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68066 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 100, success, outsize: 120 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.730+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.730+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.730+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.730+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.730+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68155 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 102, success, outsize: 120 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.743+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.743+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.743+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.743+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.743+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68247 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 104, success, outsize: 120 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.757+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.757+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.757+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.757+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.757+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68345 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 106, success, outsize: 120 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.772+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.772+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.772+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.772+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.772+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68441 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 108, success, outsize: 120 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.788+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.788+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.788+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.788+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.788+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68530 2024-04-16T07:22:48.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 110, success, outsize: 120 2024-04-16T07:22:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.803+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.803+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.803+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.803+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.803+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68622 2024-04-16T07:22:48.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 112, success, outsize: 120 2024-04-16T07:22:48.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.819+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.819+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.819+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.819+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.819+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.831 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:48 smithi104 ceph-mon[36163]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68720 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.839+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 114, success, outsize: 120 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.839+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.839+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.839+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.839+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68815 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 116, success, outsize: 120 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.853+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.853+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.853+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.853+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.853+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68904 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.866+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 118, success, outsize: 120 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.866+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.866+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.866+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.866+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68996 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 120, success, outsize: 120 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.882+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.882+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.882+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.882+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.882+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:48.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 69094 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 122, success, outsize: 120 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.896+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.896+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.896+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.896+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:22:48.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:48.896+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:49.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:49.008+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:49.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:48 smithi026 ceph-mon[28805]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:49.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.296+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:49.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:49.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:49.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.297+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:49.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.297+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:49.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:49.297+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:50.008+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:50.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:50.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:50.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:50.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:50.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:50.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:50.297+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:51.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:51.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:51.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:51.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:51.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:51.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:51.008+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: Adjusting osd_memory_target on smithi104 to 2581M 2024-04-16T07:22:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:50 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:22:51.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:51.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T07:22:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: Adjusting osd_memory_target on smithi104 to 2581M 2024-04-16T07:22:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:22:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:50 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:51.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:51.298+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:52.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:52.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:52.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:52.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:52.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:52.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:52.008+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:52.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:51 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:22:52.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:51 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:22:52.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:51 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:22:52.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:51 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:22:52.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:51 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:22:52.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:51 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:22:52.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:51 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:22:52.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:51 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:22:52.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:52.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:22:52.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:22:52.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:22:52.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 9) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004480 con 0x55fc3f51baa0 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.297+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.298+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 13 ==== client_session(renewcaps seq 9) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0047d0 con 0x55fc3f51baa0 2024-04-16T07:22:52.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:52.298+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 9) v1 from mds.0 2024-04-16T07:22:53.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.008+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:53.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.008+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:53.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.008+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:53.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.009+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:53.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.009+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:53.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:53.009+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:52 smithi104 ceph-mon[36163]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:52 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:22:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:52 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:22:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:52 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:52 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:52 smithi026 ceph-mon[28805]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:52 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:22:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:52 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:22:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:52 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:52 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.298+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.298+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.298+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.298+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.299+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:53.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:53.299+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:54.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:54.009+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:53 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:53 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:54.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:54.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:54.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:54.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:54.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:54.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:54.299+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:55.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:55.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:55.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:55.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:55.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:55.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:55.010+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:54 smithi104 ceph-mon[36163]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:54 smithi104 ceph-mon[36163]: Reconfiguring node-exporter.smithi026 (dependencies changed)... 2024-04-16T07:22:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:54 smithi104 ceph-mon[36163]: Reconfiguring daemon node-exporter.smithi026 on smithi026 2024-04-16T07:22:55.175 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:54 smithi026 ceph-mon[28805]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:55.175 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:54 smithi026 ceph-mon[28805]: Reconfiguring node-exporter.smithi026 (dependencies changed)... 2024-04-16T07:22:55.175 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:54 smithi026 ceph-mon[28805]: Reconfiguring daemon node-exporter.smithi026 on smithi026 2024-04-16T07:22:55.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:55.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:55.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:55.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:55.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:55.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:55.299+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:56.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:56.009+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:56.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:55 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:56.121 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:55 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:56.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:56.299+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:57.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:57.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:57.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:57.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:57.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:57.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:57.009+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:56 smithi104 ceph-mon[36163]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:56 smithi026 ceph-mon[28805]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:57.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.299+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:57.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:57.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:57.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.300+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:57.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.300+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:57.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:57.300+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:58.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:58.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:58.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:58.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:58.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:58.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:58.010+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:58.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:58.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:58.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:58.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:58.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:58.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:58.300+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:58 smithi104 ceph-mon[36163]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:58 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:58 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:58 smithi104 ceph-mon[36163]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-04-16T07:22:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:22:58 smithi104 ceph-mon[36163]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-04-16T07:22:58.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:58 smithi026 ceph-mon[28805]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:22:58.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:58 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:58.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:58 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:22:58.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:58 smithi026 ceph-mon[28805]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-04-16T07:22:58.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:22:58 smithi026 ceph-mon[28805]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-04-16T07:22:59.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:22:59.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:22:59.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:22:59.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:22:59.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:22:59.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:22:59.010+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:22:59.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:22:59.300+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:00.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:00.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:00.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:00.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:00.010+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:00.279 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:00 smithi026 ceph-mon[28805]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:00.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:00.300+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:00.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:00 smithi104 ceph-mon[36163]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:01.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:01.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:01.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:01.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:01.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:01.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:01.010+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:01.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:01.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:01.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:01.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:01.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:01.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:01.301+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:02.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:02.011+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:02.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:02.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:02.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:02.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:02.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:02.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:02.301+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:02.312 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:02 smithi026 ceph-mon[28805]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:02.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:02 smithi104 ceph-mon[36163]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:03.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:03.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:03.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:03.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:03.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:03.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:03.011+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:03.301+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.011+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.011+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.011+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.011+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.011+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 10) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:23:04.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.012+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.012+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.012+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.012+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.012+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.013+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 14 ==== client_session(renewcaps seq 10) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:23:04.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:04.013+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 10) v1 from mds.0 2024-04-16T07:23:04.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:04.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:04.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:04.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:04.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:04.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:04.301+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:04.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:04 smithi104 ceph-mon[36163]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:04.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:04 smithi026 ceph-mon[28805]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:05.012+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:05.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:05.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:05.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:05.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:05.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:05.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:05.302+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:05.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:05 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:05.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:05 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:05.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:05 smithi104 ceph-mon[36163]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-04-16T07:23:05.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:05 smithi104 ceph-mon[36163]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-04-16T07:23:05.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:05 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:05.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:05 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:05.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:05 smithi026 ceph-mon[28805]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-04-16T07:23:05.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:05 smithi026 ceph-mon[28805]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-04-16T07:23:06.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:06.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:06.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:06.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:06.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:06.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:06.012+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:06.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:06.302+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:06.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:06 smithi026 ceph-mon[28805]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:06.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:06 smithi104 ceph-mon[36163]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:07.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:07.013+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:07.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:07.302+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 -- 172.21.15.26:0/1026501379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 msgr2=0x7fb20c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 --2- 172.21.15.26:0/1026501379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c0fe070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8004940 tx=0x7fb1f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 -- 172.21.15.26:0/1026501379 shutdown_connections 2024-04-16T07:23:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 --2- 172.21.15.26:0/1026501379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c0fe070 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 --2- 172.21.15.26:0/1026501379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 0x7fb20c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 -- 172.21.15.26:0/1026501379 >> 172.21.15.26:0/1026501379 conn(0x7fb20c0f8540 msgr2=0x7fb20c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.710+0000 7fb214000700 1 -- 172.21.15.26:0/1026501379 shutdown_connections 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.711+0000 7fb214000700 1 -- 172.21.15.26:0/1026501379 wait complete. 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 Processor -- start 2024-04-16T07:23:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 -- start start 2024-04-16T07:23:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 0x7fb20c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c070950 con 0x7fb20c0fd710 2024-04-16T07:23:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.712+0000 7fb214000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c070ac0 con 0x7fb20c0fcdc0 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56382/0 (socket says 172.21.15.26:56382) 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 -- 172.21.15.26:0/1887339812 learned_addr learned my addr 172.21.15.26:0/1887339812 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb211600700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 0x7fb20c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 -- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 msgr2=0x7fb20c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 0x7fb20c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 -- 172.21.15.26:0/1887339812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8005040 con 0x7fb20c0fd710 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.713+0000 7fb210c00700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800b3b0 tx=0x7fb1f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.714+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f800e040 con 0x7fb20c0fd710 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.714+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1f800bcc0 con 0x7fb20c0fd710 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.714+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f801c630 con 0x7fb20c0fd710 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.714+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c070d40 con 0x7fb20c0fd710 2024-04-16T07:23:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.714+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c115ac0 con 0x7fb20c0fd710 2024-04-16T07:23:07.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.715+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c05f5f0 con 0x7fb20c0fd710 2024-04-16T07:23:07.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.717+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fb1f80139f0 con 0x7fb20c0fd710 2024-04-16T07:23:07.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.718+0000 7fb209600700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 0x7fb1f406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:07.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.718+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb1f808b840 con 0x7fb20c0fd710 2024-04-16T07:23:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.723+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fb1f8054ea0 con 0x7fb20c0fd710 2024-04-16T07:23:07.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.729+0000 7fb211600700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 0x7fb1f406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:07.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.744+0000 7fb211600700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 0x7fb1f406e1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb200006fd0 tx=0x7fb200008040 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:07.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.974+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb20c102440 con 0x7fb1f406bcf0 2024-04-16T07:23:07.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.978+0000 7fb209600700 1 -- 172.21.15.26:0/1887339812 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fb20c102440 con 0x7fb1f406bcf0 2024-04-16T07:23:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 msgr2=0x7fb1f406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 0x7fb1f406e1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb200006fd0 tx=0x7fb200008040 comp rx=0 tx=0).stop 2024-04-16T07:23:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 msgr2=0x7fb20c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800b3b0 tx=0x7fb1f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 shutdown_connections 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fb1f406bcf0 0x7fb1f406e1a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb20c0fcdc0 0x7fb20c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 --2- 172.21.15.26:0/1887339812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb20c0fd710 0x7fb20c070310 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 >> 172.21.15.26:0/1887339812 conn(0x7fb20c0f8540 msgr2=0x7fb20c0f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 shutdown_connections 2024-04-16T07:23:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:07.981+0000 7fb214000700 1 -- 172.21.15.26:0/1887339812 wait complete. 2024-04-16T07:23:07.997 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:23:08.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:08.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:08.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:08.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:08.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:08.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:08.013+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:08.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.145+0000 7fc331800700 1 -- 172.21.15.26:0/708169338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c0709a0 msgr2=0x7fc32c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.145+0000 7fc331800700 1 --2- 172.21.15.26:0/708169338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c0709a0 0x7fc32c070e10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc318004940 tx=0x7fc318009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:08.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.145+0000 7fc331800700 1 -- 172.21.15.26:0/708169338 shutdown_connections 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.146+0000 7fc331800700 1 --2- 172.21.15.26:0/708169338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c0709a0 0x7fc32c070e10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.146+0000 7fc331800700 1 --2- 172.21.15.26:0/708169338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c071f50 0x7fc32c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.146+0000 7fc331800700 1 -- 172.21.15.26:0/708169338 >> 172.21.15.26:0/708169338 conn(0x7fc32c0f8530 msgr2=0x7fc32c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.146+0000 7fc331800700 1 -- 172.21.15.26:0/708169338 shutdown_connections 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.146+0000 7fc331800700 1 -- 172.21.15.26:0/708169338 wait complete. 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 Processor -- start 2024-04-16T07:23:08.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 -- start start 2024-04-16T07:23:08.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c0709a0 0x7fc32c10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c110820 con 0x7fc32c071f50 2024-04-16T07:23:08.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc331800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c111010 con 0x7fc32c0709a0 2024-04-16T07:23:08.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.147+0000 7fc32a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:08.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38922/0 (socket says 172.21.15.26:38922) 2024-04-16T07:23:08.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 -- 172.21.15.26:0/1162470310 learned_addr learned my addr 172.21.15.26:0/1162470310 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:08.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32aa00700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c0709a0 0x7fc32c10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 -- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c0709a0 msgr2=0x7fc32c10fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c0709a0 0x7fc32c10fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 -- 172.21.15.26:0/1162470310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc318005040 con 0x7fc32c071f50 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.148+0000 7fc32a000700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc318009150 tx=0x7fc318009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.149+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31800e040 con 0x7fc32c071f50 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.149+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc31800bbf0 con 0x7fc32c071f50 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.149+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31801c630 con 0x7fc32c071f50 2024-04-16T07:23:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.149+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c111290 con 0x7fc32c071f50 2024-04-16T07:23:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.149+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c111850 con 0x7fc32c071f50 2024-04-16T07:23:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.150+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c1095c0 con 0x7fc32c071f50 2024-04-16T07:23:08.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.152+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fc31800bd60 con 0x7fc32c071f50 2024-04-16T07:23:08.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.152+0000 7fc322a00700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 0x7fc31006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.153+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc31808be20 con 0x7fc32c071f50 2024-04-16T07:23:08.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.153+0000 7fc32aa00700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 0x7fc31006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:08.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.154+0000 7fc32aa00700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 0x7fc31006e1a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006fd0 tx=0x7fc31c008040 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:08.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.157+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc318059080 con 0x7fc32c071f50 2024-04-16T07:23:08.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:08.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:08.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:08.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:08.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:08.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:08.302+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.444+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc32c111b80 con 0x7fc31006bcf0 2024-04-16T07:23:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.447+0000 7fc322a00700 1 -- 172.21.15.26:0/1162470310 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fc32c111b80 con 0x7fc31006bcf0 2024-04-16T07:23:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 msgr2=0x7fc31006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 0x7fc31006e1a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006fd0 tx=0x7fc31c008040 comp rx=0 tx=0).stop 2024-04-16T07:23:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 msgr2=0x7fc32c110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc318009150 tx=0x7fc318009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 shutdown_connections 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc31006bcf0 0x7fc31006e1a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc32c0709a0 0x7fc32c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 --2- 172.21.15.26:0/1162470310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc32c071f50 0x7fc32c110190 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 >> 172.21.15.26:0/1162470310 conn(0x7fc32c0f8530 msgr2=0x7fc32c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.450+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 shutdown_connections 2024-04-16T07:23:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.451+0000 7fc331800700 1 -- 172.21.15.26:0/1162470310 wait complete. 2024-04-16T07:23:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:08 smithi026 ceph-mon[28805]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 -- 172.21.15.26:0/1207096686 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 msgr2=0x7f0310103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 --2- 172.21.15.26:0/1207096686 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310103830 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0300004940 tx=0x7f0300009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 -- 172.21.15.26:0/1207096686 shutdown_connections 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 --2- 172.21.15.26:0/1207096686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03101046d0 0x7f0310104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 --2- 172.21.15.26:0/1207096686 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310103830 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 -- 172.21.15.26:0/1207096686 >> 172.21.15.26:0/1207096686 conn(0x7f0310072640 msgr2=0x7f0310074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.610+0000 7f0316600700 1 -- 172.21.15.26:0/1207096686 shutdown_connections 2024-04-16T07:23:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.611+0000 7f0316600700 1 -- 172.21.15.26:0/1207096686 wait complete. 2024-04-16T07:23:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.611+0000 7f0316600700 1 Processor -- start 2024-04-16T07:23:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f0316600700 1 -- start start 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f0316600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f0316600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03101046d0 0x7f0310114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f0316600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310114b20 con 0x7f0310103420 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f0316600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101194c0 con 0x7f03101046d0 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f030f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f030f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38944/0 (socket says 172.21.15.26:38944) 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.612+0000 7f030f400700 1 -- 172.21.15.26:0/2202821084 learned_addr learned my addr 172.21.15.26:0/2202821084 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f030f400700 1 -- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03101046d0 msgr2=0x7f0310114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f030f400700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03101046d0 0x7f0310114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f030f400700 1 -- 172.21.15.26:0/2202821084 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300005040 con 0x7f0310103420 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f030f400700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f030000b3b0 tx=0x7f0300009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f030000e040 con 0x7f0310103420 2024-04-16T07:23:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f030000bcc0 con 0x7f0310103420 2024-04-16T07:23:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03101196c0 con 0x7f0310103420 2024-04-16T07:23:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f030001c630 con 0x7f0310103420 2024-04-16T07:23:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.613+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310119c30 con 0x7f0310103420 2024-04-16T07:23:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.615+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031004c710 con 0x7f0310103420 2024-04-16T07:23:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.616+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f0300013980 con 0x7f0310103420 2024-04-16T07:23:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.617+0000 7f0307400700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 0x7f02f406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.617+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f030008b730 con 0x7f0310103420 2024-04-16T07:23:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.618+0000 7f030ea00700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 0x7f02f406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.619+0000 7f030ea00700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 0x7f02f406e1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f02fc006fd0 tx=0x7f02fc006db0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.622+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f0300058940 con 0x7f0310103420 2024-04-16T07:23:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:08 smithi104 ceph-mon[36163]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.865+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0310108970 con 0x7f02f406bd00 2024-04-16T07:23:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.895+0000 7f0307400700 1 -- 172.21.15.26:0/2202821084 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f0310108970 con 0x7f02f406bd00 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 starting - - - - 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (10m) 32s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (9m) 26s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (8m) 32s ago 10m 74.0M - 9.4.7 954c08fa6188 9eb0a18c75bd 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (3m) 32s ago 3m 11.7M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (3m) 32s ago 3m 22.0M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:23:08.894 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (3m) 26s ago 3m 11.8M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (3m) 26s ago 3m 11.8M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:9283 running (12m) 32s ago 12m 335M - 17.2.7-847-g45317f95 b333581ccdf5 28d81ab6bef3 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (55s) 26s ago 9m 488M - 19.0.0-2884-g218a6914 a4e4811815e4 c49f0b0c525f 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (12m) 32s ago 12m 48.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (9m) 26s ago 9m 35.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 starting - - - - 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (9m) 26s ago 9m 7650k - 1.5.0 0da6a335fe13 41dcdb8dd182 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (8m) 32s ago 8m 45.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (7m) 32s ago 7m 48.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (6m) 32s ago 6m 45.6M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (6m) 26s ago 6m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (5m) 26s ago 5m 42.9M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (5m) 26s ago 5m 42.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:23:08.895 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (8m) 32s ago 9m 50.9M - 2.43.0 a07b618ecd1d 15653c478b1e 2024-04-16T07:23:08.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 msgr2=0x7f02f406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 0x7f02f406e1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f02fc006fd0 tx=0x7f02fc006db0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 msgr2=0x7f0310113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f030000b3b0 tx=0x7f0300009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 shutdown_connections 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f02f406bd00 0x7f02f406e1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0310103420 0x7f0310113f50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 --2- 172.21.15.26:0/2202821084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03101046d0 0x7f0310114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.898+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 >> 172.21.15.26:0/2202821084 conn(0x7f0310072640 msgr2=0x7f0310073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:08.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.899+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 shutdown_connections 2024-04-16T07:23:08.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:08.899+0000 7f0316600700 1 -- 172.21.15.26:0/2202821084 wait complete. 2024-04-16T07:23:09.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:09.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:09.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:09.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:09.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:09.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:09.013+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 -- 172.21.15.26:0/2453086681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c680ff020 msgr2=0x7f2c680ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/2453086681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c680ff020 0x7f2c680ff430 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2c58004940 tx=0x7f2c58009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 -- 172.21.15.26:0/2453086681 shutdown_connections 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/2453086681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c681002d0 0x7f2c68100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/2453086681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c680ff020 0x7f2c680ff430 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 -- 172.21.15.26:0/2453086681 >> 172.21.15.26:0/2453086681 conn(0x7f2c680fa950 msgr2=0x7f2c680fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 -- 172.21.15.26:0/2453086681 shutdown_connections 2024-04-16T07:23:09.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.043+0000 7f2c6ca00700 1 -- 172.21.15.26:0/2453086681 wait complete. 2024-04-16T07:23:09.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 Processor -- start 2024-04-16T07:23:09.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 -- start start 2024-04-16T07:23:09.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 0x7f2c6810fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68110720 con 0x7f2c681002d0 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c6ca00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c681150c0 con 0x7f2c680ff020 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c65400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c65400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38968/0 (socket says 172.21.15.26:38968) 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.044+0000 7f2c65400700 1 -- 172.21.15.26:0/3559864643 learned_addr learned my addr 172.21.15.26:0/3559864643 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65e00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 0x7f2c6810fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65400700 1 -- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 msgr2=0x7f2c6810fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65400700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 0x7f2c6810fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65400700 1 -- 172.21.15.26:0/3559864643 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58005040 con 0x7f2c681002d0 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65e00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 0x7f2c6810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:23:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c65400700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400b790 tx=0x7f2c54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c5400f770 con 0x7f2c681002d0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.045+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c5400fda0 con 0x7f2c681002d0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.046+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c68115320 con 0x7f2c681002d0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.046+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c5400d310 con 0x7f2c681002d0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.046+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c68115890 con 0x7f2c681002d0 2024-04-16T07:23:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.047+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6804c710 con 0x7f2c681002d0 2024-04-16T07:23:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.048+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f2c54010420 con 0x7f2c681002d0 2024-04-16T07:23:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.049+0000 7f2c5e000700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 0x7f2c4c06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.049+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f2c5408aa60 con 0x7f2c681002d0 2024-04-16T07:23:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.049+0000 7f2c65e00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 0x7f2c4c06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.051+0000 7f2c65e00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 0x7f2c4c06e1b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800b7a0 tx=0x7f2c58009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.054+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f2c54057be0 con 0x7f2c681002d0 2024-04-16T07:23:09.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:09.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:09.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:09.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:09.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:09.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:09.303+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.382+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2c68107b90 con 0x7f2c681002d0 2024-04-16T07:23:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.383+0000 7f2c5e000700 1 -- 172.21.15.26:0/3559864643 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f2c54057770 con 0x7f2c681002d0 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:09.382 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 13, 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:23:09.383 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 msgr2=0x7f2c4c06e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 0x7f2c4c06e1b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800b7a0 tx=0x7f2c58009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 msgr2=0x7f2c68110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400b790 tx=0x7f2c54009250 comp rx=0 tx=0).stop 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 shutdown_connections 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f2c4c06bd00 0x7f2c4c06e1b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c680ff020 0x7f2c6810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 --2- 172.21.15.26:0/3559864643 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c681002d0 0x7f2c68110090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 >> 172.21.15.26:0/3559864643 conn(0x7f2c680fa950 msgr2=0x7f2c680fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 shutdown_connections 2024-04-16T07:23:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.386+0000 7f2c6ca00700 1 -- 172.21.15.26:0/3559864643 wait complete. 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 -- 172.21.15.26:0/3583397012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 msgr2=0x7f6b3c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 --2- 172.21.15.26:0/3583397012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c102340 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6b28004940 tx=0x7f6b28009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 -- 172.21.15.26:0/3583397012 shutdown_connections 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 --2- 172.21.15.26:0/3583397012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c102340 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 --2- 172.21.15.26:0/3583397012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c0ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.533+0000 7f6b40a00700 1 -- 172.21.15.26:0/3583397012 >> 172.21.15.26:0/3583397012 conn(0x7f6b3c0f7450 msgr2=0x7f6b3c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.534+0000 7f6b40a00700 1 -- 172.21.15.26:0/3583397012 shutdown_connections 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.534+0000 7f6b40a00700 1 -- 172.21.15.26:0/3583397012 wait complete. 2024-04-16T07:23:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.534+0000 7f6b40a00700 1 Processor -- start 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.534+0000 7f6b40a00700 1 -- start start 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b40a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c10dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b40a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b40a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c10e870 con 0x7f6b3c0fff20 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b40a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c113210 con 0x7f6b3c066f60 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b39400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b39400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38996/0 (socket says 172.21.15.26:38996) 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.535+0000 7f6b39400700 1 -- 172.21.15.26:0/2778347144 learned_addr learned my addr 172.21.15.26:0/2778347144 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:09.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39400700 1 -- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 msgr2=0x7f6b3c10dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39e00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c10dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39400700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c10dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39400700 1 -- 172.21.15.26:0/2778347144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b28005040 con 0x7f6b3c0fff20 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39e00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c10dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b39400700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b3b0 tx=0x7f6b28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b2800e040 con 0x7f6b3c0fff20 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b28004290 con 0x7f6b3c0fff20 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b2801c630 con 0x7f6b3c0fff20 2024-04-16T07:23:09.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.536+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c1133b0 con 0x7f6b3c0fff20 2024-04-16T07:23:09.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:09 smithi026 ceph-mon[28805]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:09.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:09 smithi026 ceph-mon[28805]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:09.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.537+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c113920 con 0x7f6b3c0fff20 2024-04-16T07:23:09.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.538+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c04c710 con 0x7f6b3c0fff20 2024-04-16T07:23:09.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.539+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f6b2800bbf0 con 0x7f6b3c0fff20 2024-04-16T07:23:09.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.540+0000 7f6b32000700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 0x7f6b2006e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.540+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f6b2808bf30 con 0x7f6b3c0fff20 2024-04-16T07:23:09.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.541+0000 7f6b39e00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 0x7f6b2006e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.542+0000 7f6b39e00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 0x7f6b2006e1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0041d0 tx=0x7f6b2c009040 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:09.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.547+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f6b280554f0 con 0x7f6b3c0fff20 2024-04-16T07:23:09.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.830+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6b3c0676c0 con 0x7f6b3c0fff20 2024-04-16T07:23:09.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.831+0000 7f6b32000700 1 -- 172.21.15.26:0/2778347144 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f6b28058bb0 con 0x7f6b3c0fff20 2024-04-16T07:23:09.830 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:23:09.830 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:23:09.830 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:23:09.830 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:23:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:23:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:23:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:09.833 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:09.833 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:09.833 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:09.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:09 smithi104 ceph-mon[36163]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:09.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:09 smithi104 ceph-mon[36163]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 msgr2=0x7f6b2006e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 0x7f6b2006e1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0041d0 tx=0x7f6b2c009040 comp rx=0 tx=0).stop 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 msgr2=0x7f6b3c10e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b3b0 tx=0x7f6b28009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 shutdown_connections 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f6b2006bd00 0x7f6b2006e1b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b3c066f60 0x7f6b3c10dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 --2- 172.21.15.26:0/2778347144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b3c0fff20 0x7f6b3c10e1f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 >> 172.21.15.26:0/2778347144 conn(0x7f6b3c0f7450 msgr2=0x7f6b3c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 shutdown_connections 2024-04-16T07:23:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.834+0000 7f6b40a00700 1 -- 172.21.15.26:0/2778347144 wait complete. 2024-04-16T07:23:09.838 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 -- 172.21.15.26:0/3586562660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fe0e0 msgr2=0x7f87c00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 --2- 172.21.15.26:0/3586562660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fe0e0 0x7f87c00fe550 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f87ac004940 tx=0x7f87ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 -- 172.21.15.26:0/3586562660 shutdown_connections 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 --2- 172.21.15.26:0/3586562660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fe0e0 0x7f87c00fe550 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 --2- 172.21.15.26:0/3586562660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fcef0 0x7f87c00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 -- 172.21.15.26:0/3586562660 >> 172.21.15.26:0/3586562660 conn(0x7f87c00f8530 msgr2=0x7f87c00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:09.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 -- 172.21.15.26:0/3586562660 shutdown_connections 2024-04-16T07:23:09.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.998+0000 7f87c8600700 1 -- 172.21.15.26:0/3586562660 wait complete. 2024-04-16T07:23:09.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.999+0000 7f87c8600700 1 Processor -- start 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.999+0000 7f87c8600700 1 -- start start 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:09.999+0000 7f87c8600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c8600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 0x7f87c0110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c8600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c0110f80 con 0x7f87c00fcef0 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c8600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c01110f0 con 0x7f87c00fe0e0 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39012/0 (socket says 172.21.15.26:39012) 2024-04-16T07:23:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 -- 172.21.15.26:0/2560423828 learned_addr learned my addr 172.21.15.26:0/2560423828 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5200700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 0x7f87c0110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 -- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 msgr2=0x7f87c0110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 0x7f87c0110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5c00700 1 -- 172.21.15.26:0/2560423828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac005040 con 0x7f87c00fcef0 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.000+0000 7f87c5200700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 0x7f87c0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87c5c00700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b790 tx=0x7f87b4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b400f6f0 con 0x7f87c00fcef0 2024-04-16T07:23:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c01113d0 con 0x7f87c00fcef0 2024-04-16T07:23:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87b400fd20 con 0x7f87c00fcef0 2024-04-16T07:23:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c0111990 con 0x7f87c00fcef0 2024-04-16T07:23:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.001+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b40103a0 con 0x7f87c00fcef0 2024-04-16T07:23:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.003+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a00052e0 con 0x7f87c00fcef0 2024-04-16T07:23:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.004+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f87b400f850 con 0x7f87c00fcef0 2024-04-16T07:23:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.005+0000 7f87bd600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 0x7f87a806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.005+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f87b408a7d0 con 0x7f87c00fcef0 2024-04-16T07:23:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.006+0000 7f87c5200700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 0x7f87a806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:10.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.007+0000 7f87c5200700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 0x7f87a806e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00b7a0 tx=0x7f87ac009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:10.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.009+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f87b40578d0 con 0x7f87c00fcef0 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:10.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:10.013+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:10.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.272+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87a0000bf0 con 0x7f87a806bcf0 2024-04-16T07:23:10.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.276+0000 7f87bd600700 1 -- 172.21.15.26:0/2560423828 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f87a0000bf0 con 0x7f87a806bcf0 2024-04-16T07:23:10.274 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "", 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:23:10.275 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:23:10.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 msgr2=0x7f87a806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 0x7f87a806e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00b7a0 tx=0x7f87ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:10.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 msgr2=0x7f87c010fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b790 tx=0x7f87b4009220 comp rx=0 tx=0).stop 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 shutdown_connections 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f87a806bcf0 0x7f87a806e1a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87c00fcef0 0x7f87c010fd30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 --2- 172.21.15.26:0/2560423828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c00fe0e0 0x7f87c0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.278+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 >> 172.21.15.26:0/2560423828 conn(0x7f87c00f8530 msgr2=0x7f87c00f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.279+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 shutdown_connections 2024-04-16T07:23:10.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.279+0000 7f87c8600700 1 -- 172.21.15.26:0/2560423828 wait complete. 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:10.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:10.303+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 -- 172.21.15.26:0/2355361275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 msgr2=0x7f9180100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 --2- 172.21.15.26:0/2355361275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f9180100740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f916c004940 tx=0x7f916c009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 -- 172.21.15.26:0/2355361275 shutdown_connections 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 --2- 172.21.15.26:0/2355361275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f9180100740 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 --2- 172.21.15.26:0/2355361275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 0x7f91800ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 -- 172.21.15.26:0/2355361275 >> 172.21.15.26:0/2355361275 conn(0x7f91800fa950 msgr2=0x7f91800fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:10.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 -- 172.21.15.26:0/2355361275 shutdown_connections 2024-04-16T07:23:10.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.434+0000 7f9187600700 1 -- 172.21.15.26:0/2355361275 wait complete. 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.435+0000 7f9187600700 1 Processor -- start 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.435+0000 7f9187600700 1 -- start start 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f9187600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 0x7f918010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f9187600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f9187600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91801106b0 con 0x7f91801002d0 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f9187600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180110820 con 0x7f91800ff020 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f917fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f917fe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39034/0 (socket says 172.21.15.26:39034) 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f917fe00700 1 -- 172.21.15.26:0/3850098894 learned_addr learned my addr 172.21.15.26:0/3850098894 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.436+0000 7f9184c00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 0x7f918010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917fe00700 1 -- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 msgr2=0x7f918010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917fe00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 0x7f918010fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917fe00700 1 -- 172.21.15.26:0/3850098894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c005040 con 0x7f91801002d0 2024-04-16T07:23:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917fe00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f916c009150 tx=0x7f916c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f916c00e040 con 0x7f91801002d0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180115220 con 0x7f91801002d0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f916c00bbf0 con 0x7f91801002d0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.437+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f916c01c630 con 0x7f91801002d0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.438+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91801157e0 con 0x7f91801002d0 2024-04-16T07:23:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.439+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918004c710 con 0x7f91801002d0 2024-04-16T07:23:10.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.440+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f916c00bd60 con 0x7f91801002d0 2024-04-16T07:23:10.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.442+0000 7f917cc00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 0x7f916806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:10.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.442+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f916c08bf40 con 0x7f91801002d0 2024-04-16T07:23:10.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.444+0000 7f9184c00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 0x7f916806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:10.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.445+0000 7f9184c00700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 0x7f916806e1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9174004990 tx=0x7f9174009ee0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.447+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f916c058fb0 con 0x7f91801002d0 2024-04-16T07:23:10.551 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:10 smithi026 ceph-mon[28805]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:10.552 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:10 smithi026 ceph-mon[28805]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:10.552 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:10 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3559864643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:23:10.552 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:10 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2778347144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:23:10.552 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:10 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:10.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.778+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9180115be0 con 0x7f91801002d0 2024-04-16T07:23:10.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.779+0000 7f917cc00700 1 -- 172.21.15.26:0/3850098894 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f916c058b40 con 0x7f91801002d0 2024-04-16T07:23:10.778 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.781+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 msgr2=0x7f916806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.781+0000 7f9187600700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 0x7f916806e1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9174004990 tx=0x7f9174009ee0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 msgr2=0x7f91801100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f916c009150 tx=0x7f916c009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 shutdown_connections 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f916806bd00 0x7f916806e1b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91800ff020 0x7f918010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 --2- 172.21.15.26:0/3850098894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91801002d0 0x7f91801100b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 >> 172.21.15.26:0/3850098894 conn(0x7f91800fa950 msgr2=0x7f91800fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 shutdown_connections 2024-04-16T07:23:10.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:10.782+0000 7f9187600700 1 -- 172.21.15.26:0/3850098894 wait complete. 2024-04-16T07:23:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:10 smithi104 ceph-mon[36163]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:10 smithi104 ceph-mon[36163]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:10 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3559864643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:23:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:10 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2778347144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:23:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:10 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:11.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:11.014+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:11.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:11.303+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:12.008 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:11 smithi026 ceph-mon[28805]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:12.008 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:11 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:12.008 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:11 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:12.008 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:11 smithi026 ceph-mon[28805]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:23:12.008 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:11 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3850098894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:23:12.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:12.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:12.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:12.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:12.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:12.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:12.014+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:11 smithi104 ceph-mon[36163]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:11 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:11 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:11 smithi104 ceph-mon[36163]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:23:12.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:11 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3850098894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 10) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:23:12.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.303+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:12.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.304+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:12.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.304+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 14 ==== client_session(renewcaps seq 10) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:12.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:12.304+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 10) v1 from mds.0 2024-04-16T07:23:13.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:12 smithi026 ceph-mon[28805]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:13.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:12 smithi026 ceph-mon[28805]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:13.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:13.014+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:12 smithi104 ceph-mon[36163]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:12 smithi104 ceph-mon[36163]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:23:13.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:13.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:13.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:13.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:13.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:13.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:13.304+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:14.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:14.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:14.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:14.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:14.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:14.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:14.014+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:14.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:14.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:14.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:14.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:14.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:14.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:14.304+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:15.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:14 smithi026 ceph-mon[28805]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:15.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.014+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:15.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.014+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:15.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.014+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:15.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.014+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:15.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.015+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:15.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:15.015+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:15.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:14 smithi104 ceph-mon[36163]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:15.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:15.304+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:16.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:16.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:16.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:16.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:16.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:16.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:16.015+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:16.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:16.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:16.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:16.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:16.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:16.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:16.305+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:16 smithi026 ceph-mon[28805]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:17.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:17.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:17.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:17.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:17.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:17.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:17.015+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:17.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:16 smithi104 ceph-mon[36163]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:17.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:17.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:17.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:17.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:17.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:17.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:17.305+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:18.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:18.015+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:18.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:18.305+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:19.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:18 smithi026 ceph-mon[28805]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:19.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:18 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:19.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:18 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:19.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:19.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:19.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:19.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:19.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:19.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:19.016+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:18 smithi104 ceph-mon[36163]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:18 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:18 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:19.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:19.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:19.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:19.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:19.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:19.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:19.305+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:19 smithi104 ceph-mon[36163]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-04-16T07:23:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:19 smithi104 ceph-mon[36163]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-04-16T07:23:20.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:19 smithi026 ceph-mon[28805]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-04-16T07:23:20.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:19 smithi026 ceph-mon[28805]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:20.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:20.016+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:20.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.305+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:20.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:20.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.305+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:20.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.305+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:20.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.306+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:20.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:20.306+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:20 smithi104 ceph-mon[36163]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:21.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:20 smithi026 ceph-mon[28805]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:21.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:21.016+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:21.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:21.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:21.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:21.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:21.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:21.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:21.306+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:22.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:22.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:22.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:22.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:22.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:22.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.016+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:22.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:22.306+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:23:22.703 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:23:22.704 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:23:22.704 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:22.704 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:22.704 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:22 smithi026 ceph-mon[28805]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:22 smithi104 ceph-mon[36163]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:22.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.998+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c001d80 con 0x563ccaa15bb0 2024-04-16T07:23:22.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:22.999+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c001d80 con 0x563ccaa15bb0 2024-04-16T07:23:23.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:23.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:23.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:23.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:23.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:23.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:23.017+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:23.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:23.306+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 11) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:24.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:24.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:24.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:24.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.017+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:24.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.018+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 15 ==== client_session(renewcaps seq 11) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:23:24.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:24.018+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 11) v1 from mds.0 2024-04-16T07:23:24.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:24.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:24.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:24.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:24.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:24.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:24.307+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:25.001 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: Upgrade: Updating mgr.smithi026.tawexg 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:23:25.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:24 smithi026 ceph-mon[28805]: Deploying daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:25.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:25.017+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:25.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: Upgrade: Updating mgr.smithi026.tawexg 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:23:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:24 smithi104 ceph-mon[36163]: Deploying daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:23:25.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:25.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:25.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:25.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:25.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:25.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:25.307+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:26.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:25 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:26.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:26.017+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:26.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:25 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:26.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:26.307+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:27.014 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:26 smithi026 ceph-mon[28805]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:27.018+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:27.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:26 smithi104 ceph-mon[36163]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:27.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:27.307+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.018+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.018+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.018+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.018+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.019+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:28.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:28.019+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:28.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:28.308+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:28 smithi026 ceph-mon[28805]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:29.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:29.019+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:29.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:28 smithi104 ceph-mon[36163]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:29.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:29.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:29.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:29.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:29.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:29.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:29.308+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:30.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:30.019+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:30.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:30.308+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:30 smithi026 ceph-mon[28805]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:30 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:30 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:31.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:30 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:31.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:31.019+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:30 smithi104 ceph-mon[36163]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:30 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:30 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:30 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:31.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0001d80 con 0x55fc3f500090 2024-04-16T07:23:31.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.288+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a0001d80 con 0x55fc3f500090 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:31.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:31.308+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:32.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:32.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:32.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:32.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:32.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:32.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:32.019+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:32.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.308+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:32.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.308+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:23:32.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.308+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:23:32.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.308+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.308+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 11) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 15 ==== client_session(renewcaps seq 11) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:32.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:32.309+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 11) v1 from mds.0 2024-04-16T07:23:32.972 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:32 smithi026 ceph-mon[28805]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:33.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:33.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:33.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:33.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:33.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:33.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:33.020+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:33.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:32 smithi104 ceph-mon[36163]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:33.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:33.309+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:34.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:34.020+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:34.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:34.309+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:35.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:34 smithi026 ceph-mon[28805]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:35.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:34 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:35.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:34 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:35.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:35.020+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:34 smithi104 ceph-mon[36163]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:34 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:34 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:35.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:35.309+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:36.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:36.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:36.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:36.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:36.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:36.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:36.020+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:36.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:36.309+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:36.943 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:36 smithi026 ceph-mon[28805]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:37.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:37.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:37.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:37.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:37.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:37.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:37.020+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:36 smithi104 ceph-mon[36163]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:37.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:37.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:37.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:37.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:37.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:37.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:37.310+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.020+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.020+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.020+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.020+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.021+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:38.021+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:37 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:37 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:37 smithi026 ceph-mon[28805]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:38.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:38.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:38.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:38.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:38.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:38.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:38.310+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:37 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:37 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:37 smithi104 ceph-mon[36163]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:39.021+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:39.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:39.311+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:40.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:40.021+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: Standby manager daemon smithi026.tawexg restarted 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: Standby manager daemon smithi026.tawexg started 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/crt"}]: dispatch 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/key"}]: dispatch 2024-04-16T07:23:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:39 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.310+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.310+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.310+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.311+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:40.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:40.311+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: Standby manager daemon smithi026.tawexg restarted 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: Standby manager daemon smithi026.tawexg started 2024-04-16T07:23:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/crt"}]: dispatch 2024-04-16T07:23:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:23:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi026.tawexg/key"}]: dispatch 2024-04-16T07:23:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:39 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.954+0000 7f12ca000700 1 -- 172.21.15.26:0/1333666284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 msgr2=0x7f12c4070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.954+0000 7f12ca000700 1 --2- 172.21.15.26:0/1333666284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4070d60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f12b0004940 tx=0x7f12b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 -- 172.21.15.26:0/1333666284 shutdown_connections 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 --2- 172.21.15.26:0/1333666284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4070d60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 --2- 172.21.15.26:0/1333666284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 0x7f12c40703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 -- 172.21.15.26:0/1333666284 >> 172.21.15.26:0/1333666284 conn(0x7f12c40f8530 msgr2=0x7f12c40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 -- 172.21.15.26:0/1333666284 shutdown_connections 2024-04-16T07:23:40.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.955+0000 7f12ca000700 1 -- 172.21.15.26:0/1333666284 wait complete. 2024-04-16T07:23:40.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 Processor -- start 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 -- start start 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 0x7f12c41145a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4114c30 con 0x7f12c40708f0 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.956+0000 7f12ca000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4115420 con 0x7f12c4071ea0 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44990/0 (socket says 172.21.15.26:44990) 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 -- 172.21.15.26:0/1527423617 learned_addr learned my addr 172.21.15.26:0/1527423617 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c2a00700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 0x7f12c41145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 -- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 msgr2=0x7f12c41145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 0x7f12c41145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 -- 172.21.15.26:0/1527423617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0005040 con 0x7f12c40708f0 2024-04-16T07:23:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.957+0000 7f12c3400700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b790 tx=0x7f12b4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.958+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12b400f6f0 con 0x7f12c40708f0 2024-04-16T07:23:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.958+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12b400fd20 con 0x7f12c40708f0 2024-04-16T07:23:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.958+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12b40103a0 con 0x7f12c40708f0 2024-04-16T07:23:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.958+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c4115700 con 0x7f12c40708f0 2024-04-16T07:23:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.958+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c4115cc0 con 0x7f12c40708f0 2024-04-16T07:23:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.959+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c405f5f0 con 0x7f12c40708f0 2024-04-16T07:23:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.960+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f12b4011400 con 0x7f12c40708f0 2024-04-16T07:23:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.961+0000 7f12bb400700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 0x7f12a8070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.961+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f12b408dd20 con 0x7f12c40708f0 2024-04-16T07:23:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.962+0000 7f12c2a00700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 0x7f12a8070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.963+0000 7f12c2a00700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 0x7f12a8070ad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f12b0004f90 tx=0x7f12b0009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:40.966+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f12b405af20 con 0x7f12c40708f0 2024-04-16T07:23:41.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:41.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:41.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:41.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:41.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:41.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:41.021+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:41.129 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:41 smithi026 ceph-mon[28805]: mgrmap e25: smithi104.uqfljg(active, since 75s), standbys: smithi026.tawexg 2024-04-16T07:23:41.129 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:41 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:41.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.220+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f12c4116000 con 0x7f12a806e620 2024-04-16T07:23:41.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.224+0000 7f12bb400700 1 -- 172.21.15.26:0/1527423617 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f12c4116000 con 0x7f12a806e620 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 msgr2=0x7f12a8070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 0x7f12a8070ad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f12b0004f90 tx=0x7f12b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 msgr2=0x7f12c4114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b790 tx=0x7f12b4009220 comp rx=0 tx=0).stop 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 shutdown_connections 2024-04-16T07:23:41.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f12a806e620 0x7f12a8070ad0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12c40708f0 0x7f12c4114060 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 --2- 172.21.15.26:0/1527423617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c4071ea0 0x7f12c41145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 >> 172.21.15.26:0/1527423617 conn(0x7f12c40f8530 msgr2=0x7f12c40f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 shutdown_connections 2024-04-16T07:23:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.227+0000 7f12ca000700 1 -- 172.21.15.26:0/1527423617 wait complete. 2024-04-16T07:23:41.243 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:23:41.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:41.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:41.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:41.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:41.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:41.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:41.311+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:41 smithi104 ceph-mon[36163]: mgrmap e25: smithi104.uqfljg(active, since 75s), standbys: smithi026.tawexg 2024-04-16T07:23:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:41 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:41.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 -- 172.21.15.26:0/2277395859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4050066b80 msgr2=0x7f40500ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 --2- 172.21.15.26:0/2277395859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4050066b80 0x7f40500ffa70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4040004940 tx=0x7f4040009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 -- 172.21.15.26:0/2277395859 shutdown_connections 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 --2- 172.21.15.26:0/2277395859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40500fffb0 0x7f40501023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 --2- 172.21.15.26:0/2277395859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4050066b80 0x7f40500ffa70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 -- 172.21.15.26:0/2277395859 >> 172.21.15.26:0/2277395859 conn(0x7f40500f74e0 msgr2=0x7f40500f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.389+0000 7f4056e00700 1 -- 172.21.15.26:0/2277395859 shutdown_connections 2024-04-16T07:23:41.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.390+0000 7f4056e00700 1 -- 172.21.15.26:0/2277395859 wait complete. 2024-04-16T07:23:41.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 Processor -- start 2024-04-16T07:23:41.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 -- start start 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4050066b80 0x7f405010fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4050110880 con 0x7f40500fffb0 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.391+0000 7f4056e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4050110fc0 con 0x7f4050066b80 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45012/0 (socket says 172.21.15.26:45012) 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404f400700 1 -- 172.21.15.26:0/2682800298 learned_addr learned my addr 172.21.15.26:0/2682800298 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404fe00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4050066b80 0x7f405010fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404f400700 1 -- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4050066b80 msgr2=0x7f405010fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.392+0000 7f404f400700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4050066b80 0x7f405010fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f404f400700 1 -- 172.21.15.26:0/2682800298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040005040 con 0x7f40500fffb0 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f404f400700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c830 tx=0x7f403c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f403c0106f0 con 0x7f40500fffb0 2024-04-16T07:23:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f403c010d20 con 0x7f40500fffb0 2024-04-16T07:23:41.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f403c0113a0 con 0x7f40500fffb0 2024-04-16T07:23:41.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40501112a0 con 0x7f40500fffb0 2024-04-16T07:23:41.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.393+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4050111860 con 0x7f40500fffb0 2024-04-16T07:23:41.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.395+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405005f5f0 con 0x7f40500fffb0 2024-04-16T07:23:41.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.396+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f403c012400 con 0x7f40500fffb0 2024-04-16T07:23:41.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.397+0000 7f4047e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 0x7f4034070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.398+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f403c08de50 con 0x7f40500fffb0 2024-04-16T07:23:41.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.398+0000 7f404fe00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 0x7f4034070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.399+0000 7f404fe00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 0x7f4034070ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4040000c00 tx=0x7f4040009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:41.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.402+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f403c05b050 con 0x7f40500fffb0 2024-04-16T07:23:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.657+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4050111c40 con 0x7f403406e620 2024-04-16T07:23:41.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.660+0000 7f4047e00700 1 -- 172.21.15.26:0/2682800298 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f4050111c40 con 0x7f403406e620 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 msgr2=0x7f4034070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 0x7f4034070ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4040000c00 tx=0x7f4040009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 msgr2=0x7f4050110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c830 tx=0x7f403c004470 comp rx=0 tx=0).stop 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 shutdown_connections 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f403406e620 0x7f4034070ad0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4050066b80 0x7f405010fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 --2- 172.21.15.26:0/2682800298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40500fffb0 0x7f4050110280 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 >> 172.21.15.26:0/2682800298 conn(0x7f40500f74e0 msgr2=0x7f40500fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:41.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 shutdown_connections 2024-04-16T07:23:41.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.663+0000 7f4056e00700 1 -- 172.21.15.26:0/2682800298 wait complete. 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 -- 172.21.15.26:0/1430546580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4070890 msgr2=0x7fc5c4070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 --2- 172.21.15.26:0/1430546580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4070890 0x7fc5c4070d00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0004880 tx=0x7fc5b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 -- 172.21.15.26:0/1430546580 shutdown_connections 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 --2- 172.21.15.26:0/1430546580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4070890 0x7fc5c4070d00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 --2- 172.21.15.26:0/1430546580 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4071e50 0x7fc5c4070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 -- 172.21.15.26:0/1430546580 >> 172.21.15.26:0/1430546580 conn(0x7fc5c40f8540 msgr2=0x7fc5c40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 -- 172.21.15.26:0/1430546580 shutdown_connections 2024-04-16T07:23:41.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.823+0000 7fc5cc600700 1 -- 172.21.15.26:0/1430546580 wait complete. 2024-04-16T07:23:41.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.824+0000 7fc5cc600700 1 Processor -- start 2024-04-16T07:23:41.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.824+0000 7fc5cc600700 1 -- start start 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.824+0000 7fc5cc600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5cc600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4071e50 0x7fc5c41101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5cc600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c41107a0 con 0x7fc5c4070890 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5cc600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4110ff0 con 0x7fc5c4071e50 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45026/0 (socket says 172.21.15.26:45026) 2024-04-16T07:23:41.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 -- 172.21.15.26:0/471207972 learned_addr learned my addr 172.21.15.26:0/471207972 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9200700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4071e50 0x7fc5c41101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 -- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4071e50 msgr2=0x7fc5c41101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4071e50 0x7fc5c41101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.825+0000 7fc5c9c00700 1 -- 172.21.15.26:0/471207972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8005040 con 0x7fc5c4070890 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5c9c00700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80096b0 tx=0x7fc5b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b800e840 con 0x7fc5c4070890 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b0005040 con 0x7fc5c4070890 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5b800f040 con 0x7fc5c4070890 2024-04-16T07:23:41.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b8013df0 con 0x7fc5c4070890 2024-04-16T07:23:41.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.826+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4111620 con 0x7fc5c4070890 2024-04-16T07:23:41.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.827+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c405f5f0 con 0x7fc5c4070890 2024-04-16T07:23:41.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.832+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7fc5b8013620 con 0x7fc5c4070890 2024-04-16T07:23:41.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.833+0000 7fc5c1600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 0x7fc5ac070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:41.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.833+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc5b808dab0 con 0x7fc5c4070890 2024-04-16T07:23:41.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.834+0000 7fc5c9200700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 0x7fc5ac070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:41.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.835+0000 7fc5c9200700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 0x7fc5ac070ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0005010 tx=0x7fc5b0009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:41.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:41.840+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc5b805ab20 con 0x7fc5c4070890 2024-04-16T07:23:42.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:42.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:42.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:42.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003013, avg_latency: 0.001506, sq_sum: 1454481101312, count=2][client_metric_type: CAP_INFO cap_hits: 4 cap_misses: 2 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:42.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:42.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:42.021+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:42.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.087+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5c4111a30 con 0x7fc5ac06e620 2024-04-16T07:23:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.117+0000 7fc5c1600700 1 -- 172.21.15.26:0/471207972 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc5c4111a30 con 0x7fc5ac06e620 2024-04-16T07:23:42.116 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:23:42.116 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (38s) 5s ago 11m 14.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (11m) 5s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (9m) 7s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (31s) 5s ago 10m 70.5M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (3m) 5s ago 3m 12.0M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (3m) 5s ago 3m 22.9M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (3m) 7s ago 3m 12.1M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (3m) 7s ago 3m 12.1M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (12s) 5s ago 12m 69.3M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (88s) 7s ago 9m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 c49f0b0c525f 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (12m) 5s ago 12m 49.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (9m) 7s ago 9m 37.9M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (45s) 5s ago 11m 7616k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (21s) 7s ago 9m 6887k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:23:42.117 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (8m) 5s ago 8m 45.5M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (8m) 5s ago 8m 48.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (7m) 5s ago 7m 45.8M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (6m) 7s ago 6m 45.1M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (6m) 7s ago 6m 43.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (5m) 7s ago 5m 42.6M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:23:42.118 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (24s) 5s ago 10m 38.3M - 2.43.0 a07b618ecd1d 04babc6f7a08 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 msgr2=0x7fc5ac070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 0x7fc5ac070ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0005010 tx=0x7fc5b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 msgr2=0x7fc5c410fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80096b0 tx=0x7fc5b8009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 shutdown_connections 2024-04-16T07:23:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5ac06e620 0x7fc5ac070ad0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5c4070890 0x7fc5c410fc60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 --2- 172.21.15.26:0/471207972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c4071e50 0x7fc5c41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.120+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 >> 172.21.15.26:0/471207972 conn(0x7fc5c40f8540 msgr2=0x7fc5c40f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.121+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 shutdown_connections 2024-04-16T07:23:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.121+0000 7fc5cc600700 1 -- 172.21.15.26:0/471207972 wait complete. 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.272+0000 7fc612600700 1 -- 172.21.15.26:0/3348246396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c070890 msgr2=0x7fc60c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.272+0000 7fc612600700 1 --2- 172.21.15.26:0/3348246396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c070890 0x7fc60c070d00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8004940 tx=0x7fc5f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 -- 172.21.15.26:0/3348246396 shutdown_connections 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 --2- 172.21.15.26:0/3348246396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c070890 0x7fc60c070d00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 --2- 172.21.15.26:0/3348246396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c071e40 0x7fc60c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 -- 172.21.15.26:0/3348246396 >> 172.21.15.26:0/3348246396 conn(0x7fc60c0f8530 msgr2=0x7fc60c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 -- 172.21.15.26:0/3348246396 shutdown_connections 2024-04-16T07:23:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.273+0000 7fc612600700 1 -- 172.21.15.26:0/3348246396 wait complete. 2024-04-16T07:23:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 Processor -- start 2024-04-16T07:23:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 -- start start 2024-04-16T07:23:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c070890 0x7fc60c10fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c1107f0 con 0x7fc60c071e40 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.274+0000 7fc612600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c110ff0 con 0x7fc60c070890 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45036/0 (socket says 172.21.15.26:45036) 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 -- 172.21.15.26:0/3373960624 learned_addr learned my addr 172.21.15.26:0/3373960624 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60b400700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c070890 0x7fc60c10fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 -- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c070890 msgr2=0x7fc60c10fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c070890 0x7fc60c10fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 -- 172.21.15.26:0/3373960624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8005040 con 0x7fc60c071e40 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.275+0000 7fc60aa00700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800b3b0 tx=0x7fc5f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.276+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f800e040 con 0x7fc60c071e40 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.276+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5f8004290 con 0x7fc60c071e40 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.276+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f801c630 con 0x7fc60c071e40 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.276+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c111270 con 0x7fc60c071e40 2024-04-16T07:23:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.276+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c1117b0 con 0x7fc60c071e40 2024-04-16T07:23:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.277+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c05f5f0 con 0x7fc60c071e40 2024-04-16T07:23:42.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.278+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7fc5f800bbf0 con 0x7fc60c071e40 2024-04-16T07:23:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.279+0000 7fc603400700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 0x7fc5f0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.279+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc5f808f4e0 con 0x7fc60c071e40 2024-04-16T07:23:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.280+0000 7fc60b400700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 0x7fc5f0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.281+0000 7fc60b400700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 0x7fc5f0070ad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc60c071b80 tx=0x7fc5fc009ee0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.284+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc5f805c550 con 0x7fc60c071e40 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:42.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:42.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.606+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc60c04bed0 con 0x7fc60c071e40 2024-04-16T07:23:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.607+0000 7fc603400700 1 -- 172.21.15.26:0/3373960624 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fc5f805c0e0 con 0x7fc60c071e40 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:23:42.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:23:42.607 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:23:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 msgr2=0x7fc5f0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 0x7fc5f0070ad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc60c071b80 tx=0x7fc5fc009ee0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 msgr2=0x7fc60c1101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800b3b0 tx=0x7fc5f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 shutdown_connections 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.609+0000 7fc612600700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fc5f006e620 0x7fc5f0070ad0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc60c071b80 tx=0x7fc5fc009ee0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.610+0000 7fc612600700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc60c070890 0x7fc60c10fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.610+0000 7fc612600700 1 --2- 172.21.15.26:0/3373960624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc60c071e40 0x7fc60c1101f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.610+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 >> 172.21.15.26:0/3373960624 conn(0x7fc60c0f8530 msgr2=0x7fc60c0f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.610+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 shutdown_connections 2024-04-16T07:23:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.610+0000 7fc612600700 1 -- 172.21.15.26:0/3373960624 wait complete. 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 -- 172.21.15.26:0/2671334782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34100010 msgr2=0x7efe341023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 --2- 172.21.15.26:0/2671334782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34100010 0x7efe341023f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efe24004940 tx=0x7efe24009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 -- 172.21.15.26:0/2671334782 shutdown_connections 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 --2- 172.21.15.26:0/2671334782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34100010 0x7efe341023f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 --2- 172.21.15.26:0/2671334782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34066250 0x7efe340ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 -- 172.21.15.26:0/2671334782 >> 172.21.15.26:0/2671334782 conn(0x7efe340f74c0 msgr2=0x7efe340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:42.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 -- 172.21.15.26:0/2671334782 shutdown_connections 2024-04-16T07:23:42.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.760+0000 7efe3c800700 1 -- 172.21.15.26:0/2671334782 wait complete. 2024-04-16T07:23:42.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.761+0000 7efe3c800700 1 Processor -- start 2024-04-16T07:23:42.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe3c800700 1 -- start start 2024-04-16T07:23:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe3c800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe3c800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34100010 0x7efe341102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe3c800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe34110f70 con 0x7efe34066250 2024-04-16T07:23:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe3c800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe341110e0 con 0x7efe34100010 2024-04-16T07:23:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe39e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.762+0000 7efe39e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45048/0 (socket says 172.21.15.26:45048) 2024-04-16T07:23:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39e00700 1 -- 172.21.15.26:0/2909135102 learned_addr learned my addr 172.21.15.26:0/2909135102 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39400700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34100010 0x7efe341102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39e00700 1 -- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34100010 msgr2=0x7efe341102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39e00700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34100010 0x7efe341102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39e00700 1 -- 172.21.15.26:0/2909135102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe24005040 con 0x7efe34066250 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.763+0000 7efe39e00700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe2800b790 tx=0x7efe28009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.764+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe2800f6f0 con 0x7efe34066250 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.764+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe2800fd20 con 0x7efe34066250 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.764+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe341113c0 con 0x7efe34066250 2024-04-16T07:23:42.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.764+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe280103a0 con 0x7efe34066250 2024-04-16T07:23:42.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.764+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe34111980 con 0x7efe34066250 2024-04-16T07:23:42.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.765+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3405f5f0 con 0x7efe34066250 2024-04-16T07:23:42.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.766+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7efe28011400 con 0x7efe34066250 2024-04-16T07:23:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.767+0000 7efe2e000700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 0x7efe1c070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.767+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7efe2808ddd0 con 0x7efe34066250 2024-04-16T07:23:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.768+0000 7efe39400700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 0x7efe1c070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.769+0000 7efe39400700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 0x7efe1c070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe2400b3b0 tx=0x7efe24009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:42.775+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7efe2805af50 con 0x7efe34066250 2024-04-16T07:23:42.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83384 2024-04-16T07:23:43.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=0 2024-04-16T07:23:43.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 choose_target_mds starting with req->inode 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:23:43.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 choose_target_mds 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) is_hash=0 hash=0 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 10 client.14524 choose_target_mds from caps on inode 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 mds is 0 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 10 client.14524 send_request rebuilding request 3 for mds.0 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 encode_cap_releases enter (req: 0x563ccaa43210, mds: 0) 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 send_request set sent_stamp to 2024-04-16T07:23:43.002886+0000 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 10 client.14524 send_request client_request(unknown.0:3 getattr pAsLsXsFs #0x1 2024-04-16T07:23:43.002851+0000 caller_uid=0, caller_gid=0{6,}) v5 to mds.0 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_request(unknown.0:3 getattr pAsLsXsFs #0x1 2024-04-16T07:23:43.002851+0000 caller_uid=0, caller_gid=0{6,}) v5 -- 0x563ccaa43510 con 0x563ccaa1cfa0 2024-04-16T07:23:43.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.001+0000 7fbf94c4e300 20 client.14524 awaiting reply|forward|kick on 0x7ffce0388360 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 16 ==== client_reply(???:3 = 0 (0) Success) v1 ==== 423+0+0 (crc 0 0 0) 0x7fbf84016810 con 0x563ccaa1cfa0 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 handle_client_reply got a reply. Safe:1 tid 3 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 insert_trace from 2024-04-16T07:23:43.002886+0000 mds.0 is_target=1 is_dentry=0 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 features 0xffffffffffffffff 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref was 1 2024-04-16T07:23:43.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 get_snap_realm 0x1 0x7fbf70007a40, nref now is 2 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x1 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 hrm is_target=1 is_dentry=0 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 12, success, outsize: 120 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 12 client.14524 add_update_inode had 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) caps pAsLsXsFs 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 update_inode_file_time 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pLsXsFs(0=pLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) pLsXsFs ctime 2024-04-16T07:20:33.192545+0000 mtime 2024-04-16T07:19:43.615094+0000 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 dir hash is 2 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 12 client.14524 add_update_inode client inode change_attr: 1 , mds inodestat change_attr: 1 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 add_update_cap issued pLsXsFs -> pAsLsXsFs from mds.0 on 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 10 client.14524 marking (I_COMPLETE|I_DIR_ORDERED) on empty dir 0x1.head(faked_ino=0 nref=5 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 put_snap_realm 0x1 0x7fbf70007a40 2 -> 1 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 handle_client_reply signalling caller 0x7ffce0388360 2024-04-16T07:23:43.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 handle_client_reply awaiting kickback on tid 3 0x7fbf773fe000 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 20 client.14524 sendrecv kickback on tid 3 0x7fbf773fe000 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 20 client.14524 lat 0.000972 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 10 client.14524 _getattr result=0 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf77400640 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=6 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:23:43.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.002+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:42 smithi026 ceph-mon[28805]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:42 smithi026 ceph-mon[28805]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:42 smithi026 ceph-mon[28805]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:42 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3373960624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:23:43.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83472 2024-04-16T07:23:43.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 14, success, outsize: 120 2024-04-16T07:23:43.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.021+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.021+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.021+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.021+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.021+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 5 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 10 client.14524 _put_inode on 0x1.head(faked_ino=0 nref=6 ll_ref=7 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 2 2024-04-16T07:23:43.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:43.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.022+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:43.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83563 2024-04-16T07:23:43.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 16, success, outsize: 120 2024-04-16T07:23:43.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.043+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.043+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.043+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.043+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.043+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.057+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efe34111c00 con 0x7efe34066250 2024-04-16T07:23:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.057+0000 7efe2e000700 1 -- 172.21.15.26:0/2909135102 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7efe2805aae0 con 0x7efe34066250 2024-04-16T07:23:43.056 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:23:43.056 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:23:43.056 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:23:43.057 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:23:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:43.059 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 msgr2=0x7efe1c070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 0x7efe1c070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe2400b3b0 tx=0x7efe24009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 msgr2=0x7efe3410fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe2800b790 tx=0x7efe28009220 comp rx=0 tx=0).stop 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 shutdown_connections 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7efe1c06e620 0x7efe1c070ad0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe34066250 0x7efe3410fd70 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe2800b790 tx=0x7efe28009220 comp rx=0 tx=0).stop 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 --2- 172.21.15.26:0/2909135102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe34100010 0x7efe341102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 >> 172.21.15.26:0/2909135102 conn(0x7efe340f74c0 msgr2=0x7efe340fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 shutdown_connections 2024-04-16T07:23:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.060+0000 7efe3c800700 1 -- 172.21.15.26:0/2909135102 wait complete. 2024-04-16T07:23:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:23:43.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83662 2024-04-16T07:23:43.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 18, success, outsize: 120 2024-04-16T07:23:43.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.062+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.062+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.062+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.062+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.062+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83757 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 20, success, outsize: 120 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.078+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.078+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.078+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.078+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.078+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:42 smithi104 ceph-mon[36163]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:42 smithi104 ceph-mon[36163]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:42 smithi104 ceph-mon[36163]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:42 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3373960624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:23:43.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83845 2024-04-16T07:23:43.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 22, success, outsize: 120 2024-04-16T07:23:43.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.093+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.093+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.093+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.093+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.093+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83936 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 24, success, outsize: 120 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.110+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.110+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.110+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.110+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.110+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84033 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 26, success, outsize: 120 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.126+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.126+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.126+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.126+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.126+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84127 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 28, success, outsize: 120 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.144+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.144+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.144+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.144+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.144+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84215 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 30, success, outsize: 120 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.160+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.160+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.160+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.160+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.160+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84307 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 32, success, outsize: 120 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.177+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.177+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.177+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.177+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.177+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84405 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 34, success, outsize: 120 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.194+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.194+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.194+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.194+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.194+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.203+0000 7f3d05e00700 1 -- 172.21.15.26:0/3037170648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 msgr2=0x7f3d000fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.203+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3037170648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d000fd1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0004940 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 -- 172.21.15.26:0/3037170648 shutdown_connections 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3037170648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 0x7f3d000fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3037170648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d000fd1d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 -- 172.21.15.26:0/3037170648 >> 172.21.15.26:0/3037170648 conn(0x7f3d000f8540 msgr2=0x7f3d000fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 -- 172.21.15.26:0/3037170648 shutdown_connections 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.204+0000 7f3d05e00700 1 -- 172.21.15.26:0/3037170648 wait complete. 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 Processor -- start 2024-04-16T07:23:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 -- start start 2024-04-16T07:23:43.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 0x7f3d00110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00110860 con 0x7f3d000fcdc0 2024-04-16T07:23:43.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3d05e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00110fa0 con 0x7f3d000fd710 2024-04-16T07:23:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.205+0000 7f3cfea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45068/0 (socket says 172.21.15.26:45068) 2024-04-16T07:23:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 -- 172.21.15.26:0/3193919427 learned_addr learned my addr 172.21.15.26:0/3193919427 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfe000700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 0x7f3d00110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 -- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 msgr2=0x7f3d00110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 0x7f3d00110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 -- 172.21.15.26:0/3193919427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0005040 con 0x7f3d000fcdc0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.206+0000 7f3cfea00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b970 tx=0x7f3cf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.207+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf000e040 con 0x7f3d000fcdc0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.207+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cf0004290 con 0x7f3d000fcdc0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.207+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d00111220 con 0x7f3d000fcdc0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.207+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf001c630 con 0x7f3d000fcdc0 2024-04-16T07:23:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.207+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001117e0 con 0x7f3d000fcdc0 2024-04-16T07:23:43.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.208+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00101a60 con 0x7f3d000fcdc0 2024-04-16T07:23:43.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.209+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f3cf000bbf0 con 0x7f3d000fcdc0 2024-04-16T07:23:43.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.210+0000 7f3cf6a00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 0x7f3ce4070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.210+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f3cf008ff00 con 0x7f3d000fcdc0 2024-04-16T07:23:43.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.211+0000 7f3cfe000700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 0x7f3ce4070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:43.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.212+0000 7f3cfe000700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 0x7f3ce4070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d000fcb90 tx=0x7f3cec009ee0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84517 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.213+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 36, success, outsize: 120 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.213+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.213+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.213+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.213+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.214+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f3cf005cff0 con 0x7f3d000fcdc0 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84605 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 38, success, outsize: 120 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.226+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.226+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.226+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.226+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.226+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84696 2024-04-16T07:23:43.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 40, success, outsize: 120 2024-04-16T07:23:43.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.240+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.240+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.240+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.240+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.240+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84793 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 42, success, outsize: 120 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.254+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.254+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.254+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.254+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.254+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84887 2024-04-16T07:23:43.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.269+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 44, success, outsize: 120 2024-04-16T07:23:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.269+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.269+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.269+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.269+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84975 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 46, success, outsize: 120 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.282+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.282+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.282+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.282+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.282+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85066 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 48, success, outsize: 120 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.296+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.296+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.296+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.296+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.296+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85163 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 50, success, outsize: 120 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.310+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.310+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.310+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.310+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.310+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:43.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:43.311+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:43.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85257 2024-04-16T07:23:43.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 52, success, outsize: 120 2024-04-16T07:23:43.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.324+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.324+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.324+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.324+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.324+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85345 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 54, success, outsize: 120 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.337+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.337+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.337+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.337+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.337+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85436 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 56, success, outsize: 120 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.352+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.352+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.352+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.352+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.352+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85533 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 58, success, outsize: 120 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.367+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.367+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.367+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.367+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.367+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85627 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 60, success, outsize: 120 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.382+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.382+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.382+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.382+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.382+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85715 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 62, success, outsize: 120 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.396+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.396+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.396+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.396+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.396+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85806 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 64, success, outsize: 120 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.411+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.411+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.411+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.411+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.411+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85903 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 66, success, outsize: 120 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.427+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.427+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.427+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.427+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:43.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:43.427+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:43.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.464+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 --> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d00111b10 con 0x7f3ce406e620 2024-04-16T07:23:43.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.468+0000 7f3cf6a00700 1 -- 172.21.15.26:0/3193919427 <== mgr.24415 v2:172.21.15.104:6828/2545344580 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3d00111b10 con 0x7f3ce406e620 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr" 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:23:43.467 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:23:43.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 msgr2=0x7f3ce4070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 0x7f3ce4070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d000fcb90 tx=0x7f3cec009ee0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 msgr2=0x7f3d0010fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b970 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 shutdown_connections 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7f3ce406e620 0x7f3ce4070ad0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d000fcdc0 0x7f3d0010fd20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 --2- 172.21.15.26:0/3193919427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d000fd710 0x7f3d00110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.471+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 >> 172.21.15.26:0/3193919427 conn(0x7f3d000f8540 msgr2=0x7f3d000f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.472+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 shutdown_connections 2024-04-16T07:23:43.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.472+0000 7f3d05e00700 1 -- 172.21.15.26:0/3193919427 wait complete. 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 -- 172.21.15.26:0/4135557159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b81046d0 msgr2=0x7fe2b8104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 --2- 172.21.15.26:0/4135557159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b81046d0 0x7fe2b8104b40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4004940 tx=0x7fe2a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 -- 172.21.15.26:0/4135557159 shutdown_connections 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 --2- 172.21.15.26:0/4135557159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b81046d0 0x7fe2b8104b40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 --2- 172.21.15.26:0/4135557159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b8103420 0x7fe2b8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 -- 172.21.15.26:0/4135557159 >> 172.21.15.26:0/4135557159 conn(0x7fe2b8072640 msgr2=0x7fe2b8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 -- 172.21.15.26:0/4135557159 shutdown_connections 2024-04-16T07:23:43.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.621+0000 7fe2bd400700 1 -- 172.21.15.26:0/4135557159 wait complete. 2024-04-16T07:23:43.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.622+0000 7fe2bd400700 1 Processor -- start 2024-04-16T07:23:43.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.622+0000 7fe2bd400700 1 -- start start 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2bd400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2bd400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b81046d0 0x7fe2b8114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2bd400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b8114a10 con 0x7fe2b8103420 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2bd400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b8114b80 con 0x7fe2b81046d0 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45082/0 (socket says 172.21.15.26:45082) 2024-04-16T07:23:43.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 -- 172.21.15.26:0/2728405286 learned_addr learned my addr 172.21.15.26:0/2728405286 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 -- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b81046d0 msgr2=0x7fe2b8114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b81046d0 0x7fe2b8114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.623+0000 7fe2b6800700 1 -- 172.21.15.26:0/2728405286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a4005040 con 0x7fe2b8103420 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2b6800700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800eb40 tx=0x7fe2a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a8007cc0 con 0x7fe2b8103420 2024-04-16T07:23:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b8119580 con 0x7fe2b8103420 2024-04-16T07:23:43.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2a800f040 con 0x7fe2b8103420 2024-04-16T07:23:43.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a8009620 con 0x7fe2b8103420 2024-04-16T07:23:43.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.624+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b8119b40 con 0x7fe2b8103420 2024-04-16T07:23:43.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.625+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b804c710 con 0x7fe2b8103420 2024-04-16T07:23:43.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.626+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) v1 ==== 92571+0+0 (secure 0 0 0) 0x7fe2a8009780 con 0x7fe2b8103420 2024-04-16T07:23:43.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.627+0000 7fe2aea00700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 0x7fe29c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:23:43.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.627+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fe2a808dd80 con 0x7fe2b8103420 2024-04-16T07:23:43.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.628+0000 7fe2b5e00700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 0x7fe29c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:23:43.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.629+0000 7fe2b5e00700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 0x7fe29c070ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009fd0 tx=0x7fe2a4009e30 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:23:43.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.632+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fe2a805ae20 con 0x7fe2b8103420 2024-04-16T07:23:43.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.984+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe2b8119d60 con 0x7fe2b8103420 2024-04-16T07:23:43.984 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:43 smithi026 ceph-mon[28805]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:43.984 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:43 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2909135102' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:23:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.984+0000 7fe2aea00700 1 -- 172.21.15.26:0/2728405286 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe2a805a9b0 con 0x7fe2b8103420 2024-04-16T07:23:43.984 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:23:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 msgr2=0x7fe29c070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 0x7fe29c070ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009fd0 tx=0x7fe2a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:23:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 msgr2=0x7fe2b8113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:23:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800eb40 tx=0x7fe2a800b040 comp rx=0 tx=0).stop 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 shutdown_connections 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:6828/2545344580,v1:172.21.15.104:6829/2545344580] conn(0x7fe29c06e630 0x7fe29c070ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.987+0000 7fe2bd400700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2b8103420 0x7fe2b8113ed0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.988+0000 7fe2bd400700 1 --2- 172.21.15.26:0/2728405286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b81046d0 0x7fe2b8114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.988+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 >> 172.21.15.26:0/2728405286 conn(0x7fe2b8072640 msgr2=0x7fe2b8073c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.988+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 shutdown_connections 2024-04-16T07:23:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:23:43.988+0000 7fe2bd400700 1 -- 172.21.15.26:0/2728405286 wait complete. 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 12) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:44.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.022+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:44.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.023+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 17 ==== client_session(renewcaps seq 12) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:23:44.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:44.023+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 12) v1 from mds.0 2024-04-16T07:23:44.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:43 smithi104 ceph-mon[36163]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:44.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:43 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2909135102' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:23:44.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:44.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:44.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:44.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:44.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:44.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:44.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:44.956 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: pgmap v42: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:44.956 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2728405286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:23:44.957 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:44 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: pgmap v42: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2728405286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:23:44.959 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:44 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:45.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:45.022+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:45.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:45.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:45.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:45.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:45.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:45.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:45.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:46.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr fail", "who": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: Activating manager daemon smithi026.tawexg 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd='[{"prefix": "mgr fail", "who": "smithi104.uqfljg"}]': finished 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: mgrmap e26: smithi026.tawexg(active, starting, since 0.0164752s) 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:23:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:23:46.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:46.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:46.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:23:46.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:45 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:46.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:46.022+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:46.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:46.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:46.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: Upgrade: Need to upgrade myself (mgr.smithi104.uqfljg) 2024-04-16T07:23:46.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "mgr fail", "who": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: Activating manager daemon smithi026.tawexg 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.24415 172.21.15.104:0/2405792110' entity='mgr.smithi104.uqfljg' cmd='[{"prefix": "mgr fail", "who": "smithi104.uqfljg"}]': finished 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: mgrmap e26: smithi026.tawexg(active, starting, since 0.0164752s) 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:23:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:23:46.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:45 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:23:46.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:46.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:46.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:46.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:46.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:46.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:46.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:47.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: mgrmap e27: smithi026.tawexg(active, since 1.02484s) 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: [16/Apr/2024:07:23:45] ENGINE Bus STARTING 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: [16/Apr/2024:07:23:46] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: [16/Apr/2024:07:23:46] ENGINE Serving on http://172.21.15.26:8765 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: [16/Apr/2024:07:23:46] ENGINE Bus STARTED 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:46 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:47.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:47.022+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:47.181 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: mgrmap e27: smithi026.tawexg(active, since 1.02484s) 2024-04-16T07:23:47.181 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: [16/Apr/2024:07:23:45] ENGINE Bus STARTING 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: [16/Apr/2024:07:23:46] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: [16/Apr/2024:07:23:46] ENGINE Serving on http://172.21.15.26:8765 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: [16/Apr/2024:07:23:46] ENGINE Bus STARTED 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:47.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:46 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:47.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:47.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:48.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:48.023+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:48.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:47 smithi104 ceph-mon[36163]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:48.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:47 smithi026 ceph-mon[28805]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:48.312+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:49.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:49.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:49.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:49.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:49.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:49.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:49.023+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:49.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:48 smithi104 ceph-mon[36163]: mgrmap e28: smithi026.tawexg(active, since 3s) 2024-04-16T07:23:49.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:48 smithi026 ceph-mon[28805]: mgrmap e28: smithi026.tawexg(active, since 3s) 2024-04-16T07:23:49.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:49.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:49.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:49.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:49.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:49.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:49.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:50.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:50.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:50.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:50.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:50.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:50.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:50.023+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:50.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:49 smithi104 ceph-mon[36163]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:50.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:49 smithi026 ceph-mon[28805]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:50.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:50.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:50.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:50.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:50.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:50.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:50.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:51.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:51.023+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:51.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:51.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:51.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:51.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:51.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:51.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:51.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:51 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:51 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:51.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:51 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:51.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:51 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:52.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:52.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:52.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:52.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:52.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:52.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:52.024+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 12) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:23:52.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.314+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 16 ==== client_session(renewcaps seq 12) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:52.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:52.314+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 12) v1 from mds.0 2024-04-16T07:23:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:52 smithi104 ceph-mon[36163]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:52 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:52 smithi026 ceph-mon[28805]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:23:52.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:52 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:53.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:53.024+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:53.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:53.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:53.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:53 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:53 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:53 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:53 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:53 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:53.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:53 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:54.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:54.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:54.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:54.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:54.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:54.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:54.024+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:54.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:54.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:54.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:54.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:54.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:54.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:54.313+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:23:54.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:54 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:23:54.713 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:23:54.713 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:23:54.713 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:23:54.714 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:23:54.714 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:23:54.714 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:54 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/1017290944' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:23:55.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:55.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:55.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:55.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:55.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:55.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:55.024+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:55.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:55.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:55.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:55.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:55.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:55.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:55.314+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:55.722 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:55 smithi026 ceph-mon[28805]: mgrmap e29: smithi026.tawexg(active, since 9s), standbys: smithi104.uqfljg 2024-04-16T07:23:55.722 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:55 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:23:55.722 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:55 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:55.722 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:55 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:55 smithi104 ceph-mon[36163]: mgrmap e29: smithi026.tawexg(active, since 9s), standbys: smithi104.uqfljg 2024-04-16T07:23:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:55 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:23:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:55 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:55 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:23:56.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:56.024+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:56.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:56.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:56.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:56.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:56.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:56.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:56.315+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:56.599 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:56 smithi026 ceph-mon[28805]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:23:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:56 smithi104 ceph-mon[36163]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:57.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:57.025+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:57.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:57.314+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:58.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:58.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:58.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:58.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 31 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:58.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:58.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.025+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:58.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:58.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:58.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:58.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:58.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:58.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:58.315+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:58.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87298 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 68, success, outsize: 120 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.734+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.734+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.734+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.734+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.734+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87380 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 70, success, outsize: 120 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.763+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.763+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.763+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.763+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.763+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87465 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 72, success, outsize: 120 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.786+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.786+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.786+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.786+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.786+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87556 2024-04-16T07:23:58.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 74, success, outsize: 120 2024-04-16T07:23:58.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.816+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.816+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.816+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.816+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.816+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:23:58 smithi104 ceph-mon[36163]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:23:58.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87644 2024-04-16T07:23:58.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 76, success, outsize: 120 2024-04-16T07:23:58.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.846+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.846+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.846+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.846+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.846+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87726 2024-04-16T07:23:58.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 78, success, outsize: 120 2024-04-16T07:23:58.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.868+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.868+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.868+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.868+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.868+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87811 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 80, success, outsize: 120 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.893+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.893+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.893+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.893+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.893+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.916 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87902 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 82, success, outsize: 120 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.918+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.918+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.918+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.918+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.918+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87990 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 84, success, outsize: 120 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.941+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.941+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.941+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.941+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.941+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88072 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 86, success, outsize: 120 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.965+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.965+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.965+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.965+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.965+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:58.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88157 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 88, success, outsize: 120 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.991+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.991+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.991+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.991+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:58.991+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:23:58 smithi026 ceph-mon[28805]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88248 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 90, success, outsize: 120 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.018+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.018+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.018+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.018+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.018+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 43 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:23:59.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.025+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88336 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 92, success, outsize: 120 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.044+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.044+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.044+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.044+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.044+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88418 2024-04-16T07:23:59.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 94, success, outsize: 120 2024-04-16T07:23:59.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.066+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.066+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.066+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.066+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.066+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88503 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 96, success, outsize: 120 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.089+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.089+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.089+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.089+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.089+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88594 2024-04-16T07:23:59.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 98, success, outsize: 120 2024-04-16T07:23:59.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.114+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.114+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.114+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.114+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.114+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88682 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 100, success, outsize: 120 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.141+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.141+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.141+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.141+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.141+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88764 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 102, success, outsize: 120 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.162+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.162+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.162+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.162+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.162+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88849 2024-04-16T07:23:59.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 104, success, outsize: 120 2024-04-16T07:23:59.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.185+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.185+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.185+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.185+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.185+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88940 2024-04-16T07:23:59.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 106, success, outsize: 120 2024-04-16T07:23:59.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.208+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.208+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.208+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.208+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.208+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89028 2024-04-16T07:23:59.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 108, success, outsize: 120 2024-04-16T07:23:59.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.231+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.231+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.231+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.231+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.231+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89110 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 110, success, outsize: 120 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.254+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.254+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.254+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.254+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.254+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89195 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 112, success, outsize: 120 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.279+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.279+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.279+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.279+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.279+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89286 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 114, success, outsize: 120 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.305+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.305+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.305+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.305+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.305+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:23:59.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:23:59.315+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:23:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89374 2024-04-16T07:23:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 116, success, outsize: 120 2024-04-16T07:23:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.330+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.330+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.330+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.330+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.330+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89456 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 118, success, outsize: 120 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.352+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.352+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.352+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.352+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.352+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89541 2024-04-16T07:23:59.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 120, success, outsize: 120 2024-04-16T07:23:59.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.377+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.377+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.377+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.377+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.377+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:23:59.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89632 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 122, success, outsize: 120 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.402+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.402+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.402+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.402+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:23:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:23:59.402+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:24:00.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:00.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:00.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:00.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:00.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:00.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:00.025+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:00.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:00.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:00.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:00.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:00.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:00.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:00.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:00.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:00.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:00 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:24:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:00 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:24:01.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.025+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:01.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.025+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:01.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.025+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:01.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.025+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:01.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.026+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:01.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:01.026+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:01.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:01.315+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:01 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:24:01.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:01 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:24:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:01 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:24:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:01 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:24:02.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:02.026+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:02.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:02.315+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:02 smithi104 ceph-mon[36163]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:02 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:24:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:02 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:24:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:02 smithi026 ceph-mon[28805]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:02 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:24:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:02 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:03.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:03.026+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:03.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.315+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:03.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:03.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.315+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:03.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.315+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:03.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:03.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:03.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:03.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:03 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:24:03.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:03 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:24:04.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:03 smithi026 ceph-mon[28805]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:24:04.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:03 smithi026 ceph-mon[28805]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 13) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:04.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:04.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:04.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:04.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.026+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:04.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.027+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 18 ==== client_session(renewcaps seq 13) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:24:04.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:04.027+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 13) v1 from mds.0 2024-04-16T07:24:04.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:04.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:04.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:04.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:04.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:04.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:04.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:24:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:24:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:04 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:05.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:05.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:05.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:05.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:05.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:05.026+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:04 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:05.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:05.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:05.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:05.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:05.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:05.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:05.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:05.719 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:05 smithi026 ceph-mon[28805]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:24:05.720 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:05 smithi026 ceph-mon[28805]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:06.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:06.027+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:05 smithi104 ceph-mon[36163]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-04-16T07:24:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:05 smithi104 ceph-mon[36163]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:06.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:06.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:06.876 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:06 smithi026 ceph-mon[28805]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:07.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:07.027+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:06 smithi104 ceph-mon[36163]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:07.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:07.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:08.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:08.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:08.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:08.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:08.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:08.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:08.027+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:08.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:08.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:09.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:08 smithi026 ceph-mon[28805]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:09.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.027+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:09.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.027+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:09.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.027+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:09.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.027+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:09.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.028+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:09.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:09.028+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:09.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:08 smithi104 ceph-mon[36163]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:09.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:09.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:10.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:10.028+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:10.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:10.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:10.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:10.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:10.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:10.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:10.316+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:11.007 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:10 smithi026 ceph-mon[28805]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:11.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:11.028+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:11.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:10 smithi104 ceph-mon[36163]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:11.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:11.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:11.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:11.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:11.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:11.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:11.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:12.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:12.028+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 13) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 17 ==== client_session(renewcaps seq 13) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:12.317+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 13) v1 from mds.0 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:24:12.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:12 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:12.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:12.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:24:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:24:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:12 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:13.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:13.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:13.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:13.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:13.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:13.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:13.028+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:13.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:13.318+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:14.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:14.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:14.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:14.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:14.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:14.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:14.029+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:14.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 -- 172.21.15.26:0/3093330010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc80709a0 msgr2=0x7f8bc8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 --2- 172.21.15.26:0/3093330010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc80709a0 0x7f8bc8070e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0004940 tx=0x7f8bb0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:14.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 -- 172.21.15.26:0/3093330010 shutdown_connections 2024-04-16T07:24:14.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 --2- 172.21.15.26:0/3093330010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc80709a0 0x7f8bc8070e10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 --2- 172.21.15.26:0/3093330010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc8071f50 0x7f8bc8070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 -- 172.21.15.26:0/3093330010 >> 172.21.15.26:0/3093330010 conn(0x7f8bc80f8530 msgr2=0x7f8bc80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 -- 172.21.15.26:0/3093330010 shutdown_connections 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.160+0000 7f8bce800700 1 -- 172.21.15.26:0/3093330010 wait complete. 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.161+0000 7f8bce800700 1 Processor -- start 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.161+0000 7f8bce800700 1 -- start start 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.161+0000 7f8bce800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc80709a0 0x7f8bc810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.161+0000 7f8bce800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bce800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8110780 con 0x7f8bc8071f50 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bce800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8110fd0 con 0x7f8bc80709a0 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50536/0 (socket says 172.21.15.26:50536) 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 -- 172.21.15.26:0/2755523985 learned_addr learned my addr 172.21.15.26:0/2755523985 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:14.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bc7400700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc80709a0 0x7f8bc810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 -- 172.21.15.26:0/2755523985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc80709a0 msgr2=0x7f8bc810fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc80709a0 0x7f8bc810fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.162+0000 7f8bbea00700 1 -- 172.21.15.26:0/2755523985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb0005040 con 0x7f8bc8071f50 2024-04-16T07:24:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bbea00700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000b3b0 tx=0x7f8bb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb000e040 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bb0004290 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb001c630 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc8111250 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.163+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc8111790 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.165+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc805f5f0 con 0x7f8bc8071f50 2024-04-16T07:24:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.166+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f8bb000bbf0 con 0x7f8bc8071f50 2024-04-16T07:24:14.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.167+0000 7f8bbfe00700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 0x7f8bac070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.167+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f8bb008f5b0 con 0x7f8bc8071f50 2024-04-16T07:24:14.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.167+0000 7f8bc7400700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 0x7f8bac070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.168+0000 7f8bc7400700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 0x7f8bac070ad0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009fd0 tx=0x7f8bb8009f00 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:14.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.172+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f8bb005c520 con 0x7f8bc8071f50 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:14.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:14.317+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.422+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 --> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bc8111bd0 con 0x7f8bac06e620 2024-04-16T07:24:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.425+0000 7f8bbfe00700 1 -- 172.21.15.26:0/2755523985 <== mgr.14718 v2:172.21.15.26:6800/846866607 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f8bc8111bd0 con 0x7f8bac06e620 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 msgr2=0x7f8bac070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 0x7f8bac070ad0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009fd0 tx=0x7f8bb8009f00 comp rx=0 tx=0).stop 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 msgr2=0x7f8bc8110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000b3b0 tx=0x7f8bb0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 shutdown_connections 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f8bac06e620 0x7f8bac070ad0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bc80709a0 0x7f8bc810fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 --2- 172.21.15.26:0/2755523985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bc8071f50 0x7f8bc8110180 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.428+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 >> 172.21.15.26:0/2755523985 conn(0x7f8bc80f8530 msgr2=0x7f8bc80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.429+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 shutdown_connections 2024-04-16T07:24:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.429+0000 7f8bce800700 1 -- 172.21.15.26:0/2755523985 wait complete. 2024-04-16T07:24:14.445 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 -- 172.21.15.26:0/2758066235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fdff0 msgr2=0x7fa7a00fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 --2- 172.21.15.26:0/2758066235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fdff0 0x7fa7a00fe460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa788004940 tx=0x7fa788009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 -- 172.21.15.26:0/2758066235 shutdown_connections 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 --2- 172.21.15.26:0/2758066235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fdff0 0x7fa7a00fe460 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 --2- 172.21.15.26:0/2758066235 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fce80 0x7fa7a00fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 -- 172.21.15.26:0/2758066235 >> 172.21.15.26:0/2758066235 conn(0x7fa7a00f8580 msgr2=0x7fa7a00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 -- 172.21.15.26:0/2758066235 shutdown_connections 2024-04-16T07:24:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.594+0000 7fa7a5400700 1 -- 172.21.15.26:0/2758066235 wait complete. 2024-04-16T07:24:14.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.595+0000 7fa7a5400700 1 Processor -- start 2024-04-16T07:24:14.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.595+0000 7fa7a5400700 1 -- start start 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa7a5400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fce80 0x7fa7a010fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa7a5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa7a5400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0110650 con 0x7fa7a00fdff0 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa7a5400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a01107c0 con 0x7fa7a00fce80 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa797e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa797e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50560/0 (socket says 172.21.15.26:50560) 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa797e00700 1 -- 172.21.15.26:0/1714416508 learned_addr learned my addr 172.21.15.26:0/1714416508 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.596+0000 7fa79e800700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fce80 0x7fa7a010fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa797e00700 1 -- 172.21.15.26:0/1714416508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fce80 msgr2=0x7fa7a010fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa797e00700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fce80 0x7fa7a010fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa797e00700 1 -- 172.21.15.26:0/1714416508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788005040 con 0x7fa7a00fdff0 2024-04-16T07:24:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa797e00700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b0d0 tx=0x7fa788009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78800e070 con 0x7fa7a00fdff0 2024-04-16T07:24:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa78800b410 con 0x7fa7a00fdff0 2024-04-16T07:24:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7880125e0 con 0x7fa7a00fdff0 2024-04-16T07:24:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.597+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a01151c0 con 0x7fa7a00fdff0 2024-04-16T07:24:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.598+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a0115730 con 0x7fa7a00fdff0 2024-04-16T07:24:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.599+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a0101980 con 0x7fa7a00fdff0 2024-04-16T07:24:14.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.600+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fa78800b580 con 0x7fa7a00fdff0 2024-04-16T07:24:14.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.601+0000 7fa796a00700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 0x7fa784070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:14.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.601+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fa78808eea0 con 0x7fa7a00fdff0 2024-04-16T07:24:14.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.601+0000 7fa79e800700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 0x7fa784070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:14.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.602+0000 7fa79e800700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 0x7fa784070ae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa790009b80 tx=0x7fa79000b040 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:14.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.606+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fa78805be10 con 0x7fa7a00fdff0 2024-04-16T07:24:14.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:14 smithi104 ceph-mon[36163]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:14.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:14 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:24:14.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:14 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:24:14.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:14 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:14 smithi026 ceph-mon[28805]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:14 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:24:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:14 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:24:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:14 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:14.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.849+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 --> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7a002b3b0 con 0x7fa78406e630 2024-04-16T07:24:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.851+0000 7fa796a00700 1 -- 172.21.15.26:0/1714416508 <== mgr.14718 v2:172.21.15.26:6800/846866607 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fa7a002b3b0 con 0x7fa78406e630 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.853+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 msgr2=0x7fa784070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.853+0000 7fa7a5400700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 0x7fa784070ae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa790009b80 tx=0x7fa79000b040 comp rx=0 tx=0).stop 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.853+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 msgr2=0x7fa7a0110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.853+0000 7fa7a5400700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b0d0 tx=0x7fa788009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 shutdown_connections 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fa78406e630 0x7fa784070ae0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7a00fce80 0x7fa7a010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 --2- 172.21.15.26:0/1714416508 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a00fdff0 0x7fa7a0110050 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:14.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 >> 172.21.15.26:0/1714416508 conn(0x7fa7a00f8580 msgr2=0x7fa7a00f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:14.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 shutdown_connections 2024-04-16T07:24:14.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:14.854+0000 7fa7a5400700 1 -- 172.21.15.26:0/1714416508 wait complete. 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.013+0000 7f2273c00700 1 -- 172.21.15.26:0/1373542887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 msgr2=0x7f226c0ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.013+0000 7f2273c00700 1 --2- 172.21.15.26:0/1373542887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0ffab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2260004940 tx=0x7f2260009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 -- 172.21.15.26:0/1373542887 shutdown_connections 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 --2- 172.21.15.26:0/1373542887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 0x7f226c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 --2- 172.21.15.26:0/1373542887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0ffab0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 -- 172.21.15.26:0/1373542887 >> 172.21.15.26:0/1373542887 conn(0x7f226c0f74c0 msgr2=0x7f226c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 -- 172.21.15.26:0/1373542887 shutdown_connections 2024-04-16T07:24:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.014+0000 7f2273c00700 1 -- 172.21.15.26:0/1373542887 wait complete. 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 Processor -- start 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 -- start start 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 0x7f226c0fb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0fbf60 con 0x7f226c066270 2024-04-16T07:24:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.015+0000 7f2273c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0fc0d0 con 0x7f226c0ffff0 2024-04-16T07:24:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.016+0000 7f2271200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.016+0000 7f2271200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50584/0 (socket says 172.21.15.26:50584) 2024-04-16T07:24:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.016+0000 7f2271200700 1 -- 172.21.15.26:0/218735403 learned_addr learned my addr 172.21.15.26:0/218735403 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.016+0000 7f226be00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 0x7f226c0fb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.017+0000 7f2271200700 1 -- 172.21.15.26:0/218735403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 msgr2=0x7f226c0fb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.017+0000 7f2271200700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 0x7f226c0fb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.017+0000 7f2271200700 1 -- 172.21.15.26:0/218735403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260005040 con 0x7f226c066270 2024-04-16T07:24:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.017+0000 7f2271200700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f226000b970 tx=0x7f2260009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.017+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226000e040 con 0x7f226c066270 2024-04-16T07:24:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.018+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f226000bbf0 con 0x7f226c066270 2024-04-16T07:24:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.018+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226001c3f0 con 0x7f226c066270 2024-04-16T07:24:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.018+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0fc350 con 0x7f226c066270 2024-04-16T07:24:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.018+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c115a90 con 0x7f226c066270 2024-04-16T07:24:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.019+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c05f5f0 con 0x7f226c066270 2024-04-16T07:24:15.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.021+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f226000bd60 con 0x7f226c066270 2024-04-16T07:24:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.021+0000 7f2268c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 0x7f2254070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.022+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f226008f490 con 0x7f226c066270 2024-04-16T07:24:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.022+0000 7f226be00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 0x7f2254070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.023+0000 7f226be00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 0x7f2254070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2258007fd0 tx=0x7f2258009040 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.027+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f22600588d0 con 0x7f226c066270 2024-04-16T07:24:15.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:15.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:15.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:15.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:15.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:15.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:15.029+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:15.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.261+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 --> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f226c0f8fb0 con 0x7f225406e620 2024-04-16T07:24:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.275+0000 7f2268c00700 1 -- 172.21.15.26:0/218735403 <== mgr.14718 v2:172.21.15.26:6800/846866607 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f226c0f8fb0 con 0x7f225406e620 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (71s) 22s ago 11m 14.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (11m) 22s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (10m) 25s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (65s) 22s ago 11m 71.8M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (4m) 22s ago 4m 12.2M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (4m) 22s ago 4m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (4m) 25s ago 4m 12.4M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (4m) 25s ago 4m 12.3M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:24:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (46s) 22s ago 13m 477M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283 running (2m) 25s ago 10m 112M - 19.0.0-2884-g218a6914 a4e4811815e4 c49f0b0c525f 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (13m) 22s ago 13m 52.3M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (10m) 25s ago 10m 38.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (78s) 22s ago 11m 7411k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (54s) 25s ago 10m 7524k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (9m) 22s ago 9m 45.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (8m) 22s ago 8m 48.6M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (8m) 22s ago 7m 46.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (7m) 25s ago 7m 45.5M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (6m) 25s ago 6m 43.7M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (6m) 25s ago 6m 43.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:24:15.276 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 starting - - - - 2024-04-16T07:24:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 msgr2=0x7f2254070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 0x7f2254070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2258007fd0 tx=0x7f2258009040 comp rx=0 tx=0).stop 2024-04-16T07:24:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 msgr2=0x7f226c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f226000b970 tx=0x7f2260009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 shutdown_connections 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f225406e620 0x7f2254070ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f226c066270 0x7f226c0fd4e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 --2- 172.21.15.26:0/218735403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f226c0ffff0 0x7f226c0fb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 >> 172.21.15.26:0/218735403 conn(0x7f226c0f74c0 msgr2=0x7f226c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 shutdown_connections 2024-04-16T07:24:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.278+0000 7f2273c00700 1 -- 172.21.15.26:0/218735403 wait complete. 2024-04-16T07:24:15.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.317+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:15.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:15.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.317+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:15.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.317+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:15.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.318+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:15.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:15.318+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 -- 172.21.15.26:0/4133700462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 msgr2=0x7fe958102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 --2- 172.21.15.26:0/4133700462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe958102390 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe940004940 tx=0x7fe940009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 -- 172.21.15.26:0/4133700462 shutdown_connections 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 --2- 172.21.15.26:0/4133700462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe958102390 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 --2- 172.21.15.26:0/4133700462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 0x7fe9580ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 -- 172.21.15.26:0/4133700462 >> 172.21.15.26:0/4133700462 conn(0x7fe9580f74c0 msgr2=0x7fe9580f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:15.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 -- 172.21.15.26:0/4133700462 shutdown_connections 2024-04-16T07:24:15.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.435+0000 7fe95d600700 1 -- 172.21.15.26:0/4133700462 wait complete. 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 Processor -- start 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 -- start start 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 0x7fe9580fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580fbee0 con 0x7fe9580fff70 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.436+0000 7fe95d600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580fc050 con 0x7fe9580662f0 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe956a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 0x7fe9580fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50598/0 (socket says 172.21.15.26:50598) 2024-04-16T07:24:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 -- 172.21.15.26:0/2486851456 learned_addr learned my addr 172.21.15.26:0/2486851456 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 -- 172.21.15.26:0/2486851456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 msgr2=0x7fe9580fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 0x7fe9580fd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.437+0000 7fe94e000700 1 -- 172.21.15.26:0/2486851456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940005040 con 0x7fe9580fff70 2024-04-16T07:24:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe94e000700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b3b0 tx=0x7fe940009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94000e040 con 0x7fe9580fff70 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe940004290 con 0x7fe9580fff70 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94001c3f0 con 0x7fe9580fff70 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9580fc2d0 con 0x7fe9580fff70 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.438+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe958115a90 con 0x7fe9580fff70 2024-04-16T07:24:15.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.439+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95805f5f0 con 0x7fe9580fff70 2024-04-16T07:24:15.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.440+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fe94000bbf0 con 0x7fe9580fff70 2024-04-16T07:24:15.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.441+0000 7fe94f400700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 0x7fe93c070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.441+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fe94008f3c0 con 0x7fe9580fff70 2024-04-16T07:24:15.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.441+0000 7fe956a00700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 0x7fe93c070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.442+0000 7fe956a00700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 0x7fe93c070ad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9580fd1f0 tx=0x7fe948009da0 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.446+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fe9400588c0 con 0x7fe9580fff70 2024-04-16T07:24:15.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:15 smithi104 ceph-mon[36163]: Upgrade: Updating mgr.smithi104.uqfljg 2024-04-16T07:24:15.728 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:15 smithi104 ceph-mon[36163]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:24:15.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:15 smithi104 ceph-mon[36163]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:15.729 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:15 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:15.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.756+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe95804bed0 con 0x7fe9580fff70 2024-04-16T07:24:15.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:15 smithi026 ceph-mon[28805]: Upgrade: Updating mgr.smithi104.uqfljg 2024-04-16T07:24:15.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:15 smithi026 ceph-mon[28805]: Deploying daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:24:15.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:15 smithi026 ceph-mon[28805]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:15.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:15 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.758+0000 7fe94f400700 1 -- 172.21.15.26:0/2486851456 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fe94005bf80 con 0x7fe9580fff70 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:24:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:24:15.758 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.761+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 msgr2=0x7fe93c070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.761+0000 7fe95d600700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 0x7fe93c070ad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9580fd1f0 tx=0x7fe948009da0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.761+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 msgr2=0x7fe9580fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.761+0000 7fe95d600700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b3b0 tx=0x7fe940009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.761+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 shutdown_connections 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fe93c06e620 0x7fe93c070ad0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9580662f0 0x7fe9580fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 --2- 172.21.15.26:0/2486851456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9580fff70 0x7fe9580fb930 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 >> 172.21.15.26:0/2486851456 conn(0x7fe9580f74c0 msgr2=0x7fe9580f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 shutdown_connections 2024-04-16T07:24:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.762+0000 7fe95d600700 1 -- 172.21.15.26:0/2486851456 wait complete. 2024-04-16T07:24:15.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 -- 172.21.15.26:0/1716187313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fe0e0 msgr2=0x7f67bc0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 --2- 172.21.15.26:0/1716187313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc0fe550 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f67a4004940 tx=0x7f67a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 -- 172.21.15.26:0/1716187313 shutdown_connections 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 --2- 172.21.15.26:0/1716187313 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc0fe550 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 --2- 172.21.15.26:0/1716187313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fcef0 0x7f67bc0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 -- 172.21.15.26:0/1716187313 >> 172.21.15.26:0/1716187313 conn(0x7f67bc0f8530 msgr2=0x7f67bc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 -- 172.21.15.26:0/1716187313 shutdown_connections 2024-04-16T07:24:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.917+0000 7f67c3200700 1 -- 172.21.15.26:0/1716187313 wait complete. 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.918+0000 7f67c3200700 1 Processor -- start 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.918+0000 7f67c3200700 1 -- start start 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67c3200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67c3200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67c3200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc110730 con 0x7f67bc0fcef0 2024-04-16T07:24:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67c3200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc110f80 con 0x7f67bc0fe0e0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67b3400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50618/0 (socket says 172.21.15.26:50618) 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 -- 172.21.15.26:0/2785621486 learned_addr learned my addr 172.21.15.26:0/2785621486 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 -- 172.21.15.26:0/2785621486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fe0e0 msgr2=0x7f67bc110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc110130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.919+0000 7f67bbe00700 1 -- 172.21.15.26:0/2785621486 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a4005040 con 0x7f67bc0fcef0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67bbe00700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00ecb0 tx=0x7f67ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac007b30 con 0x7f67bc0fcef0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67ac00f040 con 0x7f67bc0fcef0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc111260 con 0x7f67bc0fcef0 2024-04-16T07:24:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac00cca0 con 0x7f67bc0fcef0 2024-04-16T07:24:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.920+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc1117a0 con 0x7f67bc0fcef0 2024-04-16T07:24:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.921+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc101a70 con 0x7f67bc0fcef0 2024-04-16T07:24:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.922+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f67ac007c90 con 0x7f67bc0fcef0 2024-04-16T07:24:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.923+0000 7f67b8c00700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 0x7f67a0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.923+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f67ac08e200 con 0x7f67bc0fcef0 2024-04-16T07:24:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.924+0000 7f67b3400700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 0x7f67a0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.924+0000 7f67b3400700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 0x7f67a0070ad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009fd0 tx=0x7f67a4009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:15.928+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f67ac05af10 con 0x7f67bc0fcef0 2024-04-16T07:24:16.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:16.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:16.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:16.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:16.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:16.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:16.029+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:16.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.200+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f67bc05f5f0 con 0x7f67bc0fcef0 2024-04-16T07:24:16.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.201+0000 7f67b8c00700 1 -- 172.21.15.26:0/2785621486 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f67ac05ad30 con 0x7f67bc0fcef0 2024-04-16T07:24:16.201 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:24:16.201 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:24:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:24:16.203 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:16.204 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 msgr2=0x7f67a0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 0x7f67a0070ad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009fd0 tx=0x7f67a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 msgr2=0x7f67bc10fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00ecb0 tx=0x7f67ac00b040 comp rx=0 tx=0).stop 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 shutdown_connections 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f67a006e620 0x7f67a0070ad0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67bc0fcef0 0x7f67bc10fbf0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.203+0000 7f67c3200700 1 --2- 172.21.15.26:0/2785621486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc0fe0e0 0x7f67bc110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.204+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 >> 172.21.15.26:0/2785621486 conn(0x7f67bc0f8530 msgr2=0x7f67bc0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.204+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 shutdown_connections 2024-04-16T07:24:16.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.204+0000 7f67c3200700 1 -- 172.21.15.26:0/2785621486 wait complete. 2024-04-16T07:24:16.206 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:24:16.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:16.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:16.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:16.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:16.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:16.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:16.318+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:16.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.360+0000 7fd3f5a00700 1 -- 172.21.15.26:0/2168188689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f00ff020 msgr2=0x7fd3f00ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.360+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/2168188689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f00ff020 0x7fd3f00ff430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0004940 tx=0x7fd3e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 -- 172.21.15.26:0/2168188689 shutdown_connections 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/2168188689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f01002d0 0x7fd3f0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/2168188689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f00ff020 0x7fd3f00ff430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 -- 172.21.15.26:0/2168188689 >> 172.21.15.26:0/2168188689 conn(0x7fd3f00fa950 msgr2=0x7fd3f00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 -- 172.21.15.26:0/2168188689 shutdown_connections 2024-04-16T07:24:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 -- 172.21.15.26:0/2168188689 wait complete. 2024-04-16T07:24:16.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.361+0000 7fd3f5a00700 1 Processor -- start 2024-04-16T07:24:16.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3f5a00700 1 -- start start 2024-04-16T07:24:16.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3f5a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3f5a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f01002d0 0x7fd3f0110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3f5a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f0110600 con 0x7fd3f00ff020 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3f5a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f0110770 con 0x7fd3f01002d0 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3eea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3eea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50638/0 (socket says 172.21.15.26:50638) 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3eea00700 1 -- 172.21.15.26:0/3049987388 learned_addr learned my addr 172.21.15.26:0/3049987388 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.362+0000 7fd3ee000700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f01002d0 0x7fd3f0110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3eea00700 1 -- 172.21.15.26:0/3049987388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f01002d0 msgr2=0x7fd3f0110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3eea00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f01002d0 0x7fd3f0110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3eea00700 1 -- 172.21.15.26:0/3049987388 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0005040 con 0x7fd3f00ff020 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3eea00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000b0d0 tx=0x7fd3e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e000e070 con 0x7fd3f00ff020 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3e000b410 con 0x7fd3f00ff020 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e00125e0 con 0x7fd3f00ff020 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.363+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f0115170 con 0x7fd3f00ff020 2024-04-16T07:24:16.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.364+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f0115690 con 0x7fd3f00ff020 2024-04-16T07:24:16.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.365+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f004c710 con 0x7fd3f00ff020 2024-04-16T07:24:16.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.366+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fd3e000b580 con 0x7fd3f00ff020 2024-04-16T07:24:16.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.367+0000 7fd3e6a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 0x7fd3d4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.367+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fd3e008f350 con 0x7fd3f00ff020 2024-04-16T07:24:16.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.367+0000 7fd3ee000700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 0x7fd3d4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.368+0000 7fd3ee000700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 0x7fd3d4070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009b80 tx=0x7fd3dc00b040 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:16.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.370+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fd3e005c2c0 con 0x7fd3f00ff020 2024-04-16T07:24:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.610+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 --> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3f01045c0 con 0x7fd3d406e630 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.612+0000 7fd3e6a00700 1 -- 172.21.15.26:0/3049987388 <== mgr.14718 v2:172.21.15.26:6800/846866607 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fd3f01045c0 con 0x7fd3d406e630 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr" 2024-04-16T07:24:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:24:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T07:24:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T07:24:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:24:16.612 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:24:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 msgr2=0x7fd3d4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 0x7fd3d4070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009b80 tx=0x7fd3dc00b040 comp rx=0 tx=0).stop 2024-04-16T07:24:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 msgr2=0x7fd3f010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000b0d0 tx=0x7fd3e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 shutdown_connections 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fd3d406e630 0x7fd3d4070ae0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3f00ff020 0x7fd3f010fac0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 --2- 172.21.15.26:0/3049987388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3f01002d0 0x7fd3f0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.615+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 >> 172.21.15.26:0/3049987388 conn(0x7fd3f00fa950 msgr2=0x7fd3f00fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.616+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 shutdown_connections 2024-04-16T07:24:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.616+0000 7fd3f5a00700 1 -- 172.21.15.26:0/3049987388 wait complete. 2024-04-16T07:24:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:16 smithi026 ceph-mon[28805]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:16 smithi026 ceph-mon[28805]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:16 smithi026 ceph-mon[28805]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:16.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:16 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2486851456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:16.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:16 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/2785621486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:24:16.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3613944032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fce40 msgr2=0x7fc3dc0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3613944032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc0fd250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc004940 tx=0x7fc3cc009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:16.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3613944032 shutdown_connections 2024-04-16T07:24:16.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3613944032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3613944032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc0fd250 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3613944032 >> 172.21.15.26:0/3613944032 conn(0x7fc3dc0f8590 msgr2=0x7fc3dc0fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.767+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3613944032 shutdown_connections 2024-04-16T07:24:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.768+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3613944032 wait complete. 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.768+0000 7fc3e0c00700 1 Processor -- start 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.768+0000 7fc3e0c00700 1 -- start start 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3e0c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc10fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3e0c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3e0c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc110780 con 0x7fc3dc0fe040 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3e0c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc110fd0 con 0x7fc3dc0fce40 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3d9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3d9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50652/0 (socket says 172.21.15.26:50652) 2024-04-16T07:24:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3d9400700 1 -- 172.21.15.26:0/3445621015 learned_addr learned my addr 172.21.15.26:0/3445621015 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.769+0000 7fc3d9e00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc10fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d9400700 1 -- 172.21.15.26:0/3445621015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fce40 msgr2=0x7fc3dc10fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d9400700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc10fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d9400700 1 -- 172.21.15.26:0/3445621015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc005040 con 0x7fc3dc0fe040 2024-04-16T07:24:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d9400700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8002a50 tx=0x7fc3c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c8007cc0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3c800f040 con 0x7fc3dc0fe040 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc1112b0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c8009640 con 0x7fc3dc0fe040 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.770+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc1117f0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.771+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc1019d0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.773+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fc3c80097a0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.774+0000 7fc3d2000700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 0x7fc3c0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.774+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fc3c808dfe0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.774+0000 7fc3d9e00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 0x7fc3c0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.775+0000 7fc3d9e00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 0x7fc3c0070ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009fd0 tx=0x7fc3cc009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:16.779+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc3c805aeb0 con 0x7fc3dc0fe040 2024-04-16T07:24:16.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:16 smithi104 ceph-mon[36163]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:16.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:16 smithi104 ceph-mon[36163]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:16.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:16 smithi104 ceph-mon[36163]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:16.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:16 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2486851456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:16.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:16 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2785621486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:24:17.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:17.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:17.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:17.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:17.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:17.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:17.029+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:17.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.086+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc3dc04bed0 con 0x7fc3dc0fe040 2024-04-16T07:24:17.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.087+0000 7fc3d2000700 1 -- 172.21.15.26:0/3445621015 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc3c805aa40 con 0x7fc3dc0fe040 2024-04-16T07:24:17.086 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:24:17.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 msgr2=0x7fc3c0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:17.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 0x7fc3c0070ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009fd0 tx=0x7fc3cc009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:17.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 msgr2=0x7fc3dc110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:17.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8002a50 tx=0x7fc3c80046b0 comp rx=0 tx=0).stop 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 shutdown_connections 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7fc3c006e620 0x7fc3c0070ad0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3dc0fce40 0x7fc3dc10fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 --2- 172.21.15.26:0/3445621015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc3dc0fe040 0x7fc3dc110180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.090+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 >> 172.21.15.26:0/3445621015 conn(0x7fc3dc0f8590 msgr2=0x7fc3dc0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.091+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 shutdown_connections 2024-04-16T07:24:17.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:17.091+0000 7fc3e0c00700 1 -- 172.21.15.26:0/3445621015 wait complete. 2024-04-16T07:24:17.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:17.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:17.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:17.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:17.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:17.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:17.319+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:17.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:17 smithi026 ceph-mon[28805]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:17.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:17 smithi026 ceph-mon[28805]: from='client.? 172.21.15.26:0/3445621015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:24:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:17 smithi104 ceph-mon[36163]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:17.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:17 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/3445621015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:24:18.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.029+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:18.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.029+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:18.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.029+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:18.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.029+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:18.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.030+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:18.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:18.030+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:18.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:18.319+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:18.731 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:18 smithi104 ceph-mon[36163]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:18 smithi026 ceph-mon[28805]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:19.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:19.030+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:19.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:19.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:19.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:19.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:19.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:19.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:19.319+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:20.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:20.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:20.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:20.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:20.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:20.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:20.030+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.319+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:20.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:20.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:20 smithi104 ceph-mon[36163]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:20 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:20 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:20 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:20.708 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:20 smithi026 ceph-mon[28805]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:20.708 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:20 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:20.708 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:20 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:20.708 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:20 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:21.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:21.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:21.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:21.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:21.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:21.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:21.030+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:21.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:21.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:22.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.030+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:22.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.030+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:22.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.030+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:22.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.030+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:22.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.031+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:22.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:22.031+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:22.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:22.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:22.511 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:22 smithi104 ceph-mon[36163]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:22.733 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:22 smithi026 ceph-mon[28805]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:23.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:23.031+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:23.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:23.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:24.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:24.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:24:24.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:24:24.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 14) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.031+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.032+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 19 ==== client_session(renewcaps seq 14) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:24:24.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:24.032+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 14) v1 from mds.0 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:24.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:24.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:24.663 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:24 smithi026 ceph-mon[28805]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:24.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:24 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:24.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:24 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:24.831 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:24 smithi104 ceph-mon[36163]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:24.831 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:24 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:24 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:25.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:25.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:25.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:25.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:25.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:25.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:25.031+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:25.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:25.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:25.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:25.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:25.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:25.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:25.320+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:26.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:26.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:26.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:26.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:26.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:26.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:26.031+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:26.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:26.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:26.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:26.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:26.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:26.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:26.321+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:26 smithi104 ceph-mon[36163]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:26 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:26 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:26 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:26 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:26 smithi026 ceph-mon[28805]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:26 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:26 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:26 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:26 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:27.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:27.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:27.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:27.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:27.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:27.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:27.032+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:27.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:27.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:27.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:27.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:27.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:27.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:27.321+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:28.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:28.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:28.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:28.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:28.032+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:28.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:28.321+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:28 smithi026 ceph-mon[28805]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:28 smithi104 ceph-mon[36163]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:29.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:29.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:29.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:29.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:29.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:29.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:29.032+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:29.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73810 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 124, success, outsize: 120 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.095+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.095+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.095+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.095+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.095+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73888 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.122+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 126, success, outsize: 120 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.122+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.122+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.122+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.122+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73969 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 128, success, outsize: 120 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.150+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.150+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.150+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.150+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.150+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74056 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.179+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 130, success, outsize: 120 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.179+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.179+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.179+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.179+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74140 2024-04-16T07:24:29.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.208+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 132, success, outsize: 120 2024-04-16T07:24:29.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.208+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.208+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.208+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.208+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74218 2024-04-16T07:24:29.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.234+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 134, success, outsize: 120 2024-04-16T07:24:29.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.234+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.234+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.234+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.234+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74299 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 136, success, outsize: 120 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.260+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.260+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.260+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.260+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.260+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74386 2024-04-16T07:24:29.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.288+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 138, success, outsize: 120 2024-04-16T07:24:29.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.288+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.288+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.288+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.288+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74470 2024-04-16T07:24:29.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 140, success, outsize: 120 2024-04-16T07:24:29.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.316+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.316+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.316+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.316+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.316+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 68 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.321+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:29.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74548 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.344+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 142, success, outsize: 120 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.344+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.344+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.344+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.344+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74629 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.371+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 144, success, outsize: 120 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.371+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.371+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.371+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.371+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74716 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 146, success, outsize: 120 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.400+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.400+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.400+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.400+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.400+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74800 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.429+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 148, success, outsize: 120 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.429+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.429+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.429+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.429+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74878 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 150, success, outsize: 120 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.450+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.450+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.450+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.450+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.450+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74959 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 152, success, outsize: 120 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.475+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.475+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.475+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.475+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.475+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75046 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.499+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 154, success, outsize: 120 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.499+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.499+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.499+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.499+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75130 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 156, success, outsize: 120 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.524+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.524+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.524+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.524+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.524+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75208 2024-04-16T07:24:29.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 158, success, outsize: 120 2024-04-16T07:24:29.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.543+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.543+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.543+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.543+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.543+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75289 2024-04-16T07:24:29.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 160, success, outsize: 120 2024-04-16T07:24:29.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.563+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.563+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.563+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.563+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.563+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75376 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 162, success, outsize: 120 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.581+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.581+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.581+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.581+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.581+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75460 2024-04-16T07:24:29.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 164, success, outsize: 120 2024-04-16T07:24:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.602+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.602+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.602+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.602+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.602+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75538 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 166, success, outsize: 120 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.622+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.622+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.622+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.622+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.622+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75619 2024-04-16T07:24:29.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 168, success, outsize: 120 2024-04-16T07:24:29.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.642+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.642+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.642+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.642+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.642+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75706 2024-04-16T07:24:29.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 170, success, outsize: 120 2024-04-16T07:24:29.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.662+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.662+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.662+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.662+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.662+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75790 2024-04-16T07:24:29.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 172, success, outsize: 120 2024-04-16T07:24:29.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.682+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.682+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.682+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.682+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.682+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75868 2024-04-16T07:24:29.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 174, success, outsize: 120 2024-04-16T07:24:29.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.701+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.701+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.701+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.701+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.701+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75949 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 176, success, outsize: 120 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.719+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.719+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.719+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.719+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.719+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76036 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 178, success, outsize: 120 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.739+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.739+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.739+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.739+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:24:29.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:29.739+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:24:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:24:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:24:29.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:24:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:24:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:24:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:29 smithi026 ceph-mon[28805]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:24:29.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:29 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/274399146' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.032+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.032+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.032+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.032+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.033+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:30.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:30.033+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:30.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.321+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:30.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.321+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:30.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:30.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.322+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:30.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.322+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:30.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:30.322+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:30 smithi026 ceph-mon[28805]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:30 smithi026 ceph-mon[28805]: mgrmap e30: smithi026.tawexg(active, since 44s), standbys: smithi104.uqfljg 2024-04-16T07:24:30.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:30 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:30.831 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:30 smithi104 ceph-mon[36163]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:30 smithi104 ceph-mon[36163]: mgrmap e30: smithi026.tawexg(active, since 44s), standbys: smithi104.uqfljg 2024-04-16T07:24:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:30 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:31.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:31.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:31.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:31.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:31.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:31.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:31.033+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:31.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:31.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:31.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:31.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:31.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:31.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:31.322+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi026.tawexg"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi026.tawexg"}]': finished 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.uqfljg"}]: dispatch 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.uqfljg"}]': finished 2024-04-16T07:24:31.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:31 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi026.tawexg"}]: dispatch 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi026.tawexg"}]': finished 2024-04-16T07:24:31.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.uqfljg"}]: dispatch 2024-04-16T07:24:31.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.uqfljg"}]': finished 2024-04-16T07:24:31.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:31 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:32.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:32.033+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 14) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:32.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:32.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:24:32.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:32.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.322+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:32.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.323+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 18 ==== client_session(renewcaps seq 14) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:32.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:32.323+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 14) v1 from mds.0 2024-04-16T07:24:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:32 smithi104 ceph-mon[36163]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:32 smithi104 ceph-mon[36163]: Upgrade: Setting container_image for all mgr 2024-04-16T07:24:32.961 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:32 smithi026 ceph-mon[28805]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:32.961 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:32 smithi026 ceph-mon[28805]: Upgrade: Setting container_image for all mgr 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:33.033+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:33.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.322+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:33.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:33.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.322+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:33.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.322+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:33.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.323+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:33.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:33.323+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:34.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.033+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:34.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.033+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:34.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.033+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:34.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.033+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:34.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.034+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:34.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:34.034+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:34.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:34.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:34.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:34.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:34.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:34.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:34.323+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:34.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:34.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: Upgrade: Updating mon.smithi026 2024-04-16T07:24:34.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:34.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:24:34.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:24:34.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:34.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:34 smithi026 ceph-mon[28805]: Deploying daemon mon.smithi026 on smithi026 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: Upgrade: Updating mon.smithi026 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:24:34.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:34 smithi104 ceph-mon[36163]: Deploying daemon mon.smithi026 on smithi026 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:35.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:35.034+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:35.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:35.323+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:36.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:36.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:36.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:36.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:36.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:36.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.034+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:36 smithi026 systemd[1]: Stopping Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:36.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:36.323+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.451+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 msgr2=0x563ccaa1ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 msgr2=0x563ccaa1ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf8001b420 tx=0x7fbf8001c8b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf8001b420 tx=0x7fbf8001c8b0 comp rx=0 tx=0).stop 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 msgr2=0x563ccaa1ab70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:24:36.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563ccaa15bb0 0x563ccaa1ab70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:36.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf7000a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:36.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:36.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000d870 con 0x7fbf70008390 2024-04-16T07:24:36.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000d9e0 con 0x563ccaa01090 2024-04-16T07:24:36.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.452+0000 7fbf77400640 0 client.14524 ms_handle_reset on v2:172.21.15.26:3300/0 2024-04-16T07:24:36.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.454+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf7000a770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:24:36.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.454+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf7000a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:24:36.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.454+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:24:36.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.454+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:36.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.455+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf7000a770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:24:36.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.455+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf7000a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:36.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.456+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mdsmap=16+,monmap=3+}) v3 -- 0x7fbf5c001d80 con 0x563ccaa01090 2024-04-16T07:24:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.458+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-16T07:24:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.458+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00a5c0 tx=0x7fbf8c005af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:36.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.461+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c0105e0 con 0x563ccaa01090 2024-04-16T07:24:36.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:36.462+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf8c010c10 con 0x563ccaa01090 2024-04-16T07:24:36.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:36 smithi104 ceph-mon[36163]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:36.584 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:36 smithi026 ceph-mon[28805]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:36.585 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:36 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[28800]: 2024-04-16T07:24:36.449+0000 7fec66400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi026 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:24:36.585 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:36 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[28800]: 2024-04-16T07:24:36.449+0000 7fec66400700 -1 mon.smithi026@0(leader) e2 *** Got Signal Terminated *** 2024-04-16T07:24:37.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:36 smithi026 podman[92041]: 2024-04-16 07:24:36.75886972 +0000 UTC m=+0.447951137 container died 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:37.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:37.034+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:37.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:37.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:37.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:37.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:37.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:37.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:37.324+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:37.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 podman[92041]: 2024-04-16 07:24:37.241739251 +0000 UTC m=+0.930820662 container cleanup 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T07:24:37.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 bash[92041]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026 2024-04-16T07:24:37.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 podman[92066]: 2024-04-16 07:24:37.42593962 +0000 UTC m=+0.653962472 container remove 32be14f27f0623ffd96c1f3a3ff56f3cafb692d3ca2f48cf022026389368bb0f (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409) 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.034+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.034+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.034+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.034+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.035+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:38.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:38.035+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service: Deactivated successfully. 2024-04-16T07:24:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 systemd[1]: Stopped Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:24:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:37 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service: Consumed 25.954s CPU time. 2024-04-16T07:24:38.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:38.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:38.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:38.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:38.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:38.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:38.324+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:38.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:38 smithi026 systemd[1]: Starting Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:24:39.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:39.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:39.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:39.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:39.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:39.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:39.035+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:39.081 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:38 smithi026 podman[92195]: 2024-04-16 07:24:38.679692334 +0000 UTC m=+0.044512387 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:24:39.081 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:38 smithi026 podman[92195]: 2024-04-16 07:24:38.819272362 +0000 UTC m=+0.184092431 container create 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-04-16T07:24:39.081 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 podman[92195]: 2024-04-16 07:24:39.081508227 +0000 UTC m=+0.446328303 container init 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:39.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:39.324+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:39.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 podman[92195]: 2024-04-16 07:24:39.094152158 +0000 UTC m=+0.458972215 container start 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: pidfile_write: ignore empty --pid-file 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: load: jerasure load: lrc 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: RocksDB version: 7.9.2 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Git sha 0 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: DB SUMMARY 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: DB Session ID: X9NQ5Y7260ZZ04AIJBNB 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: CURRENT file: CURRENT 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T07:24:39.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi026/store.db dir, Total Num: 1, files: 000033.sst 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi026/store.db: 000031.log size: 1224625 ; 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.error_if_exists: 0 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.create_if_missing: 0 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.paranoid_checks: 1 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.env: 0x55d610fc1cc0 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.info_log: 0x55d621911ae0 2024-04-16T07:24:39.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.statistics: (nil) 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.use_fsync: 0 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_log_file_size: 0 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T07:24:39.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_fallocate: 1 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.use_direct_reads: 0 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T07:24:39.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.db_log_dir: 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.wal_dir: 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.write_buffer_manager: 0x55d6218e21e0 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T07:24:39.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.unordered_write: 0 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T07:24:39.517 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.row_cache: None 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.wal_filter: None 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.two_write_queues: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.wal_compression: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.atomic_flush: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T07:24:39.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.log_readahead_size: 0 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_background_jobs: 2 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_background_compactions: -1 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_subcompactions: 1 2024-04-16T07:24:39.519 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_open_files: -1 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T07:24:39.520 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_background_flushes: -1 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Compression algorithms supported: 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kZSTD supported: 0 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kXpressCompression supported: 0 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kZlibCompression supported: 1 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kSnappyCompression supported: 1 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kLZ4Compression supported: 1 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: kBZip2Compression supported: 0 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000015 2024-04-16T07:24:39.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.merge_operator: 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_filter: None 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d621911cc0) 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-04-16T07:24:39.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_type: 0 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_index_type: 0 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_shortening: 1 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: checksum: 4 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: no_block_cache: 0 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache: 0x55d621915090 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_options: 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: capacity : 536870912 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: num_shard_bits : 4 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: strict_capacity_limit : 0 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-04-16T07:24:39.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_compressed: (nil) 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: persistent_cache: (nil) 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size: 4096 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size_deviation: 10 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_restart_interval: 16 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_block_restart_interval: 1 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: metadata_block_size: 4096 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: partition_filters: 0 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: use_delta_encoding: 1 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: filter_policy: bloomfilter 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: whole_key_filtering: 1 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: verify_compression: 0 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: format_version: 5 2024-04-16T07:24:39.524 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_index_compression: 1 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_align: 0 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: max_auto_readahead_size: 262144 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: prepopulate_block_cache: 0 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: initial_auto_readahead_size: 8192 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression: NoCompression 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T07:24:39.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.num_levels: 7 2024-04-16T07:24:39.526 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T07:24:39.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T07:24:39.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T07:24:39.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T07:24:39.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.inplace_update_support: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.bloom_locality: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.max_successive_merges: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T07:24:39.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.ttl: 2592000 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enable_blob_files: false 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.min_blob_size: 0 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T07:24:39.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11834, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ec0f0823-6219-414d-9944-e406e9cc67c4 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252279154436, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252279162601, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1112637, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11835, "largest_seqno": 12486, "table_properties": {"data_size": 1108809, "index_size": 2020, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 6972, "raw_average_key_size": 23, "raw_value_size": 1102271, "raw_average_value_size": 3736, "num_data_blocks": 91, "num_entries": 295, "num_filter_entries": 295, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713252279, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ec0f0823-6219-414d-9944-e406e9cc67c4", "db_session_id": "X9NQ5Y7260ZZ04AIJBNB", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252279162783, "job": 1, "event": "recovery_finished"} 2024-04-16T07:24:39.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi026/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d621a04000 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: DB pointer 0x55d6219ee000 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** DB Stats ** 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:24:39.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: L0 1/0 1.06 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 234.5 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: L6 1/0 5.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Sum 2/0 6.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 234.5 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 234.5 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T07:24:39.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 234.5 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative compaction: 0.00 GB write, 77.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval compaction: 0.00 GB write, 77.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:24:39.536 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Block cache BinnedLRUCache@0x55d621915090#2 capacity: 512.00 MB usage: 2.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,2.08 KB,0.000396371%) Misc(1,0.00 KB,0%) 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: starting mon.smithi026 rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi026 fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???) e2 preinit fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).mds e16 new map 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).mds e16 print_map 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: e16 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: legacy client fscid: 1 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Filesystem 'cephfs' (1) 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: fs_name cephfs 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: epoch 15 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: tableserver 0 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: root 0 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: session_timeout 60 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: session_autoclose 300 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: max_file_size 1099511627776 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: max_xattr_size 65536 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: required_client_features {} 2024-04-16T07:24:39.538 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: last_failure 0 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: last_failure_osd_epoch 47 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: max_mds 1 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: in 0 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: up {0=14470} 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: failed 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: damaged 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: stopped 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_pools [3] 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: metadata_pool 2 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: inline_data disabled 2024-04-16T07:24:39.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: balancer 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: bal_rank_mask -1 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: standby_count_wanted 1 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: qdb_cluster leader: 0 members: 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: [mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Standby daemons: 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: [mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: [mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: [mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T07:24:39.540 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:24:39.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:24:39.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:24:39.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-16T07:24:39.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 bash[92195]: 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e 2024-04-16T07:24:39.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 systemd[1]: Started Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:24:40.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:40.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:40.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:40.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:40.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:40.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:40.035+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: pgmap v29: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026 calling monitor election 2024-04-16T07:24:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: monmap epoch 2 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: last_changed 2024-04-16T07:14:05.634368+0000 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: created 2024-04-16T07:10:43.696046+0000 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: min_mon_release 17 (quincy) 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: election_strategy: 1 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.smithi026 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mgrmap e30: smithi026.tawexg(active, since 54s), standbys: smithi104.uqfljg 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: overall HEALTH_OK 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: from='mgr.14718 ' entity='' 2024-04-16T07:24:40.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:39 smithi026 ceph-mon[92215]: mgrmap e31: smithi026.tawexg(active, since 54s), standbys: smithi104.uqfljg 2024-04-16T07:24:40.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.324+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:40.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:40.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.324+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:40.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.324+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:40.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.325+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:40.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:40.325+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:24:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: from='mgr.14718 172.21.15.26:0/3771544035' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:24:40.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: pgmap v29: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: mon.smithi026 calling monitor election 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: monmap epoch 2 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: last_changed 2024-04-16T07:14:05.634368+0000 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: created 2024-04-16T07:10:43.696046+0000 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: min_mon_release 17 (quincy) 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: election_strategy: 1 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.smithi026 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: mgrmap e30: smithi026.tawexg(active, since 54s), standbys: smithi104.uqfljg 2024-04-16T07:24:40.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: overall HEALTH_OK 2024-04-16T07:24:40.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: from='mgr.14718 ' entity='' 2024-04-16T07:24:40.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:39 smithi104 ceph-mon[36163]: mgrmap e31: smithi026.tawexg(active, since 54s), standbys: smithi104.uqfljg 2024-04-16T07:24:41.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:41.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:41.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:41.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:41.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:41.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:41.035+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:41.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:41.325+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:42.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.035+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:42.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.035+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:42.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.035+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:42.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.035+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:42.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.036+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:42.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:42.036+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:42.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:42.325+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:43.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:43.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:43.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:43.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:43.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:43.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:43.036+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:43.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:43.325+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 15) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:44.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:44.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:44.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:44.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.036+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:44.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.037+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 20 ==== client_session(renewcaps seq 15) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:24:44.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:44.037+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 15) v1 from mds.0 2024-04-16T07:24:44.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.325+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:44.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:44.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.325+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:44.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.325+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:44.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:44.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:44.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:45.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:45.036+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:45.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:45.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:45.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:45.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:45.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:45.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:45.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:46.036+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:46.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:46.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:46.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:46.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:46.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:46.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:46.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:47.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:47.037+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:47.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 -- 172.21.15.26:0/3401442715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 msgr2=0x7f0b800fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:47.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 --2- 172.21.15.26:0/3401442715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b800fe070 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c004940 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 -- 172.21.15.26:0/3401442715 shutdown_connections 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 --2- 172.21.15.26:0/3401442715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b800fe070 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 --2- 172.21.15.26:0/3401442715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 0x7f0b800fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 -- 172.21.15.26:0/3401442715 >> 172.21.15.26:0/3401442715 conn(0x7f0b800f8540 msgr2=0x7f0b800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 -- 172.21.15.26:0/3401442715 shutdown_connections 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.256+0000 7f0b86a00700 1 -- 172.21.15.26:0/3401442715 wait complete. 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 Processor -- start 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 -- start start 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 0x7f0b8010fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b80110890 con 0x7f0b800fd710 2024-04-16T07:24:47.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.257+0000 7f0b86a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b80110fd0 con 0x7f0b800fcdc0 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54204/0 (socket says 172.21.15.26:54204) 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 -- 172.21.15.26:0/2328544299 learned_addr learned my addr 172.21.15.26:0/2328544299 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 0x7f0b8010fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 msgr2=0x7f0b8010fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 0x7f0b8010fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.258+0000 7f0b7f400700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c005040 con 0x7f0b800fd710 2024-04-16T07:24:47.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b7f400700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b970 tx=0x7f0b6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6c00e040 con 0x7f0b800fd710 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b6c004290 con 0x7f0b800fd710 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b80111250 con 0x7f0b800fd710 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6c01c3f0 con 0x7f0b800fd710 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.259+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b80111810 con 0x7f0b800fd710 2024-04-16T07:24:47.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.261+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80101a60 con 0x7f0b800fd710 2024-04-16T07:24:47.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.261+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 31) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f0b6c00bbf0 con 0x7f0b800fd710 2024-04-16T07:24:47.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.262+0000 7f0b77e00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:47.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.262+0000 7f0b7fe00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 msgr2=0x7f0b640711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/846866607 2024-04-16T07:24:47.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.262+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f0b6c08fc90 con 0x7f0b800fd710 2024-04-16T07:24:47.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.263+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:24:47.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.268+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b6c05c520 con 0x7f0b800fd710 2024-04-16T07:24:47.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:47.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:47.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:47.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:47.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:47.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:47.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:47.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.464+0000 7f0b7fe00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 msgr2=0x7f0b640711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/846866607 2024-04-16T07:24:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.464+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:24:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.517+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b80105aa0 con 0x7f0b6406ed20 2024-04-16T07:24:47.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.865+0000 7f0b7fe00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 msgr2=0x7f0b640711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/846866607 2024-04-16T07:24:47.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:47.865+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:48.037+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:48.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:48.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:48.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:48.666+0000 7f0b7fe00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 msgr2=0x7f0b640711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/846866607 2024-04-16T07:24:48.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:48.666+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:24:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:48.853+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 32) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f0b6c08dfe0 con 0x7f0b800fd710 2024-04-16T07:24:49.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:24:49.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:24:49.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:24:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:24:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:24:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:48 smithi026 ceph-mon[92215]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:49.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:49.037+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg restarted 2024-04-16T07:24:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: Standby manager daemon smithi104.uqfljg started 2024-04-16T07:24:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/crt"}]: dispatch 2024-04-16T07:24:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T07:24:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.uqfljg/key"}]: dispatch 2024-04-16T07:24:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:48 smithi104 ceph-mon[36163]: from='mgr.? 172.21.15.104:0/3984634626' entity='mgr.smithi104.uqfljg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T07:24:49.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:49.225+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mgrmap(e 33) v1 ==== 92370+0+0 (secure 0 0 0) 0x7f0b6c058dc0 con 0x7f0b800fd710 2024-04-16T07:24:49.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:49.226+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 msgr2=0x7f0b640711d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:24:49.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:49.226+0000 7f0b77e00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:49.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:49.326+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:50.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:50.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:50.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:50.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:50.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:50.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:50.037+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.238+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mon.0 v2:172.21.15.26:3300/0 9 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f0b6c044590 con 0x7f0b800fd710 2024-04-16T07:24:50.240 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: mgrmap e32: smithi026.tawexg(active, since 64s), standbys: smithi104.uqfljg 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: Active manager daemon smithi026.tawexg restarted 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: Activating manager daemon smithi026.tawexg 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: mgrmap e33: smithi026.tawexg(active, starting, since 0.00846971s), standbys: smithi104.uqfljg 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:24:50.241 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:24:50.242 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:24:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.240+0000 7f0b77e00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 0x7f0b64029810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.240+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b80105aa0 con 0x7f0b64029440 2024-04-16T07:24:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.244+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 0x7f0b64029810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.250+0000 7f0b7fe00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 0x7f0b64029810 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b800fdbc0 tx=0x7f0b70004570 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:50.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.256+0000 7f0b77e00700 1 -- 172.21.15.26:0/2328544299 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f0b80105aa0 con 0x7f0b64029440 2024-04-16T07:24:50.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.259+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 msgr2=0x7f0b64029810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.259+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 0x7f0b64029810 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b800fdbc0 tx=0x7f0b70004570 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.259+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 msgr2=0x7f0b80110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.259+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b970 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 shutdown_connections 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b64029440 0x7f0b64029810 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:6800/846866607,v1:172.21.15.26:6801/846866607] conn(0x7f0b6406ed20 0x7f0b640711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b800fcdc0 0x7f0b8010fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 --2- 172.21.15.26:0/2328544299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b800fd710 0x7f0b80110290 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.260+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 >> 172.21.15.26:0/2328544299 conn(0x7f0b800f8540 msgr2=0x7f0b800f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.261+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 shutdown_connections 2024-04-16T07:24:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.261+0000 7f0b86a00700 1 -- 172.21.15.26:0/2328544299 wait complete. 2024-04-16T07:24:50.277 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:24:50.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:50.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:50.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:50.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:50.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:50.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:50.327+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: mgrmap e32: smithi026.tawexg(active, since 64s), standbys: smithi104.uqfljg 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: Active manager daemon smithi026.tawexg restarted 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: Activating manager daemon smithi026.tawexg 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: mgrmap e33: smithi026.tawexg(active, starting, since 0.00846971s), standbys: smithi104.uqfljg 2024-04-16T07:24:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi026.tawexg", "id": "smithi026.tawexg"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr metadata", "who": "smithi104.uqfljg", "id": "smithi104.uqfljg"}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:24:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: Manager daemon smithi026.tawexg is now available 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/mirror_snapshot_schedule"}]: dispatch 2024-04-16T07:24:50.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:49 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.tawexg/trash_purge_schedule"}]: dispatch 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 -- 172.21.15.26:0/1171006338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa264071f50 msgr2=0x7fa264070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 --2- 172.21.15.26:0/1171006338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa264071f50 0x7fa264070460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa258004940 tx=0x7fa258009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 -- 172.21.15.26:0/1171006338 shutdown_connections 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 --2- 172.21.15.26:0/1171006338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2640709a0 0x7fa264070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 --2- 172.21.15.26:0/1171006338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa264071f50 0x7fa264070460 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 -- 172.21.15.26:0/1171006338 >> 172.21.15.26:0/1171006338 conn(0x7fa2640f8530 msgr2=0x7fa2640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 -- 172.21.15.26:0/1171006338 shutdown_connections 2024-04-16T07:24:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.417+0000 7fa26be00700 1 -- 172.21.15.26:0/1171006338 wait complete. 2024-04-16T07:24:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 Processor -- start 2024-04-16T07:24:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 -- start start 2024-04-16T07:24:50.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa264071f50 0x7fa264110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264110750 con 0x7fa2640709a0 2024-04-16T07:24:50.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.418+0000 7fa26be00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264110fa0 con 0x7fa264071f50 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56168/0 (socket says 172.21.15.26:56168) 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 -- 172.21.15.26:0/3829573338 learned_addr learned my addr 172.21.15.26:0/3829573338 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa268a00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa264071f50 0x7fa264110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 -- 172.21.15.26:0/3829573338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa264071f50 msgr2=0x7fa264110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa264071f50 0x7fa264110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.419+0000 7fa269400700 1 -- 172.21.15.26:0/3829573338 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258005040 con 0x7fa2640709a0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa269400700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa25800b0d0 tx=0x7fa258009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25800e070 con 0x7fa2640709a0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa25800b410 con 0x7fa2640709a0 2024-04-16T07:24:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa264111220 con 0x7fa2640709a0 2024-04-16T07:24:50.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa258012600 con 0x7fa2640709a0 2024-04-16T07:24:50.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.420+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa264111760 con 0x7fa2640709a0 2024-04-16T07:24:50.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.422+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26405f5f0 con 0x7fa2640709a0 2024-04-16T07:24:50.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.422+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7fa25800ba60 con 0x7fa2640709a0 2024-04-16T07:24:50.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.424+0000 7fa261600700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 0x7fa24c070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.424+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa25808f250 con 0x7fa2640709a0 2024-04-16T07:24:50.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.424+0000 7fa268a00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 0x7fa24c070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.425+0000 7fa268a00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 0x7fa24c070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa250004600 tx=0x7fa25000b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:50.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.430+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa258058280 con 0x7fa2640709a0 2024-04-16T07:24:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.700+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa264111a70 con 0x7fa24c06e900 2024-04-16T07:24:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.702+0000 7fa261600700 1 -- 172.21.15.26:0/3829573338 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fa264111a70 con 0x7fa24c06e900 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 msgr2=0x7fa24c070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 0x7fa24c070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa250004600 tx=0x7fa25000b040 comp rx=0 tx=0).stop 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 msgr2=0x7fa26410fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa25800b0d0 tx=0x7fa258009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 shutdown_connections 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa24c06e900 0x7fa24c070db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2640709a0 0x7fa26410fc10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 --2- 172.21.15.26:0/3829573338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa264071f50 0x7fa264110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 >> 172.21.15.26:0/3829573338 conn(0x7fa2640f8530 msgr2=0x7fa2640f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:50.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 shutdown_connections 2024-04-16T07:24:50.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.706+0000 7fa26be00700 1 -- 172.21.15.26:0/3829573338 wait complete. 2024-04-16T07:24:50.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 -- 172.21.15.26:0/2026851046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c0fff80 msgr2=0x7f9d1c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 --2- 172.21.15.26:0/2026851046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c102360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004940 tx=0x7f9d08009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 -- 172.21.15.26:0/2026851046 shutdown_connections 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 --2- 172.21.15.26:0/2026851046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c102360 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 --2- 172.21.15.26:0/2026851046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c066b20 0x7f9d1c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.862+0000 7f9d22e00700 1 -- 172.21.15.26:0/2026851046 >> 172.21.15.26:0/2026851046 conn(0x7f9d1c0f7450 msgr2=0x7f9d1c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.863+0000 7f9d22e00700 1 -- 172.21.15.26:0/2026851046 shutdown_connections 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.863+0000 7f9d22e00700 1 -- 172.21.15.26:0/2026851046 wait complete. 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 Processor -- start 2024-04-16T07:24:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 -- start start 2024-04-16T07:24:50.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c10e5d0 con 0x7f9d1c066b20 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.864+0000 7f9d22e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c10edd0 con 0x7f9d1c0fff80 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56186/0 (socket says 172.21.15.26:56186) 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 -- 172.21.15.26:0/210704424 learned_addr learned my addr 172.21.15.26:0/210704424 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1b400700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 -- 172.21.15.26:0/210704424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c0fff80 msgr2=0x7f9d1c10dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c10dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.865+0000 7f9d1be00700 1 -- 172.21.15.26:0/210704424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d08005040 con 0x7f9d1c066b20 2024-04-16T07:24:50.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.866+0000 7f9d1be00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c8e0 tx=0x7f9d0c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:50.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.867+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0c00a760 con 0x7f9d1c066b20 2024-04-16T07:24:50.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.867+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d0c00ad90 con 0x7f9d1c066b20 2024-04-16T07:24:50.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.867+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0c0113a0 con 0x7f9d1c066b20 2024-04-16T07:24:50.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.867+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c10f0b0 con 0x7f9d1c066b20 2024-04-16T07:24:50.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.867+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c10f5f0 con 0x7f9d1c066b20 2024-04-16T07:24:50.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.869+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c05f5f0 con 0x7f9d1c066b20 2024-04-16T07:24:50.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.870+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f9d0c012400 con 0x7f9d1c066b20 2024-04-16T07:24:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.871+0000 7f9d13e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 0x7f9d00070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.871+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d0c08e3c0 con 0x7f9d1c066b20 2024-04-16T07:24:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.871+0000 7f9d1b400700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 0x7f9d00070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:50.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.872+0000 7f9d1b400700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 0x7f9d00070db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fd0 tx=0x7f9d08009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:50.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:50.876+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9d0c057470 con 0x7f9d1c066b20 2024-04-16T07:24:50.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:50 smithi026 ceph-mon[92215]: [16/Apr/2024:07:24:49] ENGINE Bus STARTING 2024-04-16T07:24:50.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:50 smithi026 ceph-mon[92215]: mgrmap e34: smithi026.tawexg(active, since 1.02912s), standbys: smithi104.uqfljg 2024-04-16T07:24:51.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:51.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:51.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:51.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:51.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:51.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:51.037+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:51.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.129+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d1c10f960 con 0x7f9d0006e900 2024-04-16T07:24:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.153+0000 7f9d13e00700 1 -- 172.21.15.26:0/210704424 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9d1c10f960 con 0x7f9d0006e900 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (107s) 24s ago 12m 14.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (12m) 24s ago 12m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (11m) 27s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (100s) 24s ago 11m 72.2M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (5m) 24s ago 5m 12.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (4m) 24s ago 4m 23.3M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (5m) 27s ago 5m 12.6M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (4m) 27s ago 4m 12.5M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (81s) 24s ago 13m 493M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:24:51.153 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (32s) 27s ago 10m 66.6M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (13m) 24s ago 14m 52.5M 2048M 17.2.7-847-g45317f95 b333581ccdf5 32be14f27f06 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (10m) 27s ago 10m 39.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 dd8dd9b5b0a8 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (114s) 24s ago 12m 7608k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (90s) 27s ago 10m 7402k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (9m) 24s ago 9m 46.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (9m) 24s ago 9m 48.9M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (8m) 24s ago 8m 46.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (8m) 27s ago 8m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (7m) 27s ago 7m 43.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (6m) 27s ago 6m 43.4M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:24:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (40s) 24s ago 11m 43.4M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:24:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.156+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 msgr2=0x7f9d00070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.156+0000 7f9d22e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 0x7f9d00070db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fd0 tx=0x7f9d08009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.156+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 msgr2=0x7f9d1c10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.156+0000 7f9d22e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c8e0 tx=0x7f9d0c0044c0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.156+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 shutdown_connections 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9d0006e900 0x7f9d00070db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d1c066b20 0x7f9d1c10da90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 --2- 172.21.15.26:0/210704424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d1c0fff80 0x7f9d1c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 >> 172.21.15.26:0/210704424 conn(0x7f9d1c0f7450 msgr2=0x7f9d1c0f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 shutdown_connections 2024-04-16T07:24:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.157+0000 7f9d22e00700 1 -- 172.21.15.26:0/210704424 wait complete. 2024-04-16T07:24:51.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.312+0000 7fe4c6e00700 1 -- 172.21.15.26:0/1193977590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 msgr2=0x7fe4c00ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.312+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/1193977590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c00ff410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0004940 tx=0x7fe4b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 -- 172.21.15.26:0/1193977590 shutdown_connections 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/1193977590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 0x7fe4c0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/1193977590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c00ff410 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 -- 172.21.15.26:0/1193977590 >> 172.21.15.26:0/1193977590 conn(0x7fe4c00fa950 msgr2=0x7fe4c00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 -- 172.21.15.26:0/1193977590 shutdown_connections 2024-04-16T07:24:51.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.313+0000 7fe4c6e00700 1 -- 172.21.15.26:0/1193977590 wait complete. 2024-04-16T07:24:51.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 Processor -- start 2024-04-16T07:24:51.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 -- start start 2024-04-16T07:24:51.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 0x7fe4c0110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c0110760 con 0x7fe4c00ff000 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.314+0000 7fe4c6e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c0110fb0 con 0x7fe4c01002b0 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56206/0 (socket says 172.21.15.26:56206) 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 -- 172.21.15.26:0/2307232045 learned_addr learned my addr 172.21.15.26:0/2307232045 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:51.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bf400700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 0x7fe4c0110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 -- 172.21.15.26:0/2307232045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 msgr2=0x7fe4c0110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 0x7fe4c0110160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.315+0000 7fe4bfe00700 1 -- 172.21.15.26:0/2307232045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0005040 con 0x7fe4c00ff000 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4bfe00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b3b0 tx=0x7fe4b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4b000e040 con 0x7fe4c00ff000 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4b000bcc0 con 0x7fe4c00ff000 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c0111230 con 0x7fe4c00ff000 2024-04-16T07:24:51.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4b001c630 con 0x7fe4c00ff000 2024-04-16T07:24:51.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.316+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c0111770 con 0x7fe4c00ff000 2024-04-16T07:24:51.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.318+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a00052e0 con 0x7fe4c00ff000 2024-04-16T07:24:51.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.319+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7fe4b0013420 con 0x7fe4c00ff000 2024-04-16T07:24:51.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.320+0000 7fe4b7e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 0x7fe4a4070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.320+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4b008f0c0 con 0x7fe4c00ff000 2024-04-16T07:24:51.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.321+0000 7fe4bf400700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 0x7fe4a4070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.322+0000 7fe4bf400700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 0x7fe4a4070db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac004c40 tx=0x7fe4ac009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:51.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.326+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe4b0058070 con 0x7fe4c00ff000 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:51.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:51.327+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:50 smithi104 ceph-mon[36163]: [16/Apr/2024:07:24:49] ENGINE Bus STARTING 2024-04-16T07:24:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:50 smithi104 ceph-mon[36163]: mgrmap e34: smithi026.tawexg(active, since 1.02912s), standbys: smithi104.uqfljg 2024-04-16T07:24:51.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.695+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe4a0005da0 con 0x7fe4c00ff000 2024-04-16T07:24:51.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.696+0000 7fe4b7e00700 1 -- 172.21.15.26:0/2307232045 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7fe4b005b730 con 0x7fe4c00ff000 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:24:51.696 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 11, 2024-04-16T07:24:51.697 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T07:24:51.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:24:51.697 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:24:51.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 msgr2=0x7fe4a4070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 0x7fe4a4070db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac004c40 tx=0x7fe4ac009040 comp rx=0 tx=0).stop 2024-04-16T07:24:51.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 msgr2=0x7fe4c010fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b3b0 tx=0x7fe4b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 shutdown_connections 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe4a406e900 0x7fe4a4070db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c00ff000 0x7fe4c010fc20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 --2- 172.21.15.26:0/2307232045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4c01002b0 0x7fe4c0110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.699+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 >> 172.21.15.26:0/2307232045 conn(0x7fe4c00fa950 msgr2=0x7fe4c00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.700+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 shutdown_connections 2024-04-16T07:24:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.700+0000 7fe4c6e00700 1 -- 172.21.15.26:0/2307232045 wait complete. 2024-04-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 -- 172.21.15.26:0/2719119956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638072820 msgr2=0x7f2638072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 --2- 172.21.15.26:0/2719119956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638072820 0x7f2638072c30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2628004940 tx=0x7f2628009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 -- 172.21.15.26:0/2719119956 shutdown_connections 2024-04-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 --2- 172.21.15.26:0/2719119956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638073170 0x7f2638073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 --2- 172.21.15.26:0/2719119956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638072820 0x7f2638072c30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 -- 172.21.15.26:0/2719119956 >> 172.21.15.26:0/2719119956 conn(0x7f26380fa950 msgr2=0x7f26380fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 -- 172.21.15.26:0/2719119956 shutdown_connections 2024-04-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.854+0000 7f263cc00700 1 -- 172.21.15.26:0/2719119956 wait complete. 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.855+0000 7f263cc00700 1 Processor -- start 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f263cc00700 1 -- start start 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f263cc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 0x7f2638114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f263cc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f263cc00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638114a20 con 0x7f2638114560 2024-04-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f263cc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638114b60 con 0x7f2638072820 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f2635400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f2635400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56252/0 (socket says 172.21.15.26:56252) 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f2635e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 0x7f2638114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.856+0000 7f2635400700 1 -- 172.21.15.26:0/2423791853 learned_addr learned my addr 172.21.15.26:0/2423791853 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f2635400700 1 -- 172.21.15.26:0/2423791853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 msgr2=0x7f2638114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f2635400700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 0x7f2638114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f2635400700 1 -- 172.21.15.26:0/2423791853 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628005040 con 0x7f2638114560 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f2635e00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 0x7f2638114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f2635400700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f262400eb40 tx=0x7f262400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.857+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2624007cc0 con 0x7f2638114560 2024-04-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.858+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f262400f040 con 0x7f2638114560 2024-04-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.858+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26381159b0 con 0x7f2638114560 2024-04-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.858+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f262400ca10 con 0x7f2638114560 2024-04-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.858+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2638115f40 con 0x7f2638114560 2024-04-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.859+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263805f5f0 con 0x7f2638114560 2024-04-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.861+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f262400cc80 con 0x7f2638114560 2024-04-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.862+0000 7f262e000700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 0x7f261c070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.862+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f262408e040 con 0x7f2638114560 2024-04-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.862+0000 7f2635e00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 0x7f261c070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.863+0000 7f2635e00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 0x7f261c070db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2628013870 tx=0x7f2628009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.869+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f26240570a0 con 0x7f2638114560 2024-04-16T07:24:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:51.880+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7f2624057470 con 0x7f2638114560 2024-04-16T07:24:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: [16/Apr/2024:07:24:50] ENGINE Serving on http://172.21.15.26:8765 2024-04-16T07:24:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: [16/Apr/2024:07:24:50] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:24:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: [16/Apr/2024:07:24:50] ENGINE Bus STARTED 2024-04-16T07:24:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:52.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:52.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:52.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:51 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2307232045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:52.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:52.038+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: [16/Apr/2024:07:24:50] ENGINE Serving on http://172.21.15.26:8765 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: [16/Apr/2024:07:24:50] ENGINE Serving on https://172.21.15.26:7150 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: [16/Apr/2024:07:24:50] ENGINE Bus STARTED 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:52.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:51 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2307232045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:24:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.160+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f263810c070 con 0x7f2638114560 2024-04-16T07:24:52.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.161+0000 7f262e000700 1 -- 172.21.15.26:0/2423791853 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f2624079780 con 0x7f2638114560 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:24:52.161 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:24:52.162 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:52.163 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.164+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 msgr2=0x7f261c070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.164+0000 7f263cc00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 0x7f261c070db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2628013870 tx=0x7f2628009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.164+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 msgr2=0x7f26381153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.164+0000 7f263cc00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f262400eb40 tx=0x7f262400b040 comp rx=0 tx=0).stop 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 shutdown_connections 2024-04-16T07:24:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f261c06e900 0x7f261c070db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2638072820 0x7f2638114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 --2- 172.21.15.26:0/2423791853 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2638114560 0x7f26381153b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 >> 172.21.15.26:0/2423791853 conn(0x7f26380fa950 msgr2=0x7f26380fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 shutdown_connections 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.165+0000 7f263cc00700 1 -- 172.21.15.26:0/2423791853 wait complete. 2024-04-16T07:24:52.165 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:24:52.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 -- 172.21.15.26:0/3083101092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fe0e0 msgr2=0x7efd5c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 --2- 172.21.15.26:0/3083101092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c0fe550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efd48004940 tx=0x7efd48009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 -- 172.21.15.26:0/3083101092 shutdown_connections 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 --2- 172.21.15.26:0/3083101092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c0fe550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 --2- 172.21.15.26:0/3083101092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fcef0 0x7efd5c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.313+0000 7efd61e00700 1 -- 172.21.15.26:0/3083101092 >> 172.21.15.26:0/3083101092 conn(0x7efd5c0f8530 msgr2=0x7efd5c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.314+0000 7efd61e00700 1 -- 172.21.15.26:0/3083101092 shutdown_connections 2024-04-16T07:24:52.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.314+0000 7efd61e00700 1 -- 172.21.15.26:0/3083101092 wait complete. 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.314+0000 7efd61e00700 1 Processor -- start 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd61e00700 1 -- start start 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd61e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd61e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd61e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c10e5d0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd61e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c10edd0 con 0x7efd5c0fe0e0 2024-04-16T07:24:52.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd5aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd5aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56262/0 (socket says 172.21.15.26:56262) 2024-04-16T07:24:52.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd5aa00700 1 -- 172.21.15.26:0/3155315195 learned_addr learned my addr 172.21.15.26:0/3155315195 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:52.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.315+0000 7efd5a000700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd5aa00700 1 -- 172.21.15.26:0/3155315195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fe0e0 msgr2=0x7efd5c10dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd5aa00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c10dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd5aa00700 1 -- 172.21.15.26:0/3155315195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd48005040 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd5aa00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd4c001ce0 tx=0x7efd4c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd4c007b30 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd4c00f040 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd4c0096d0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.316+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c10f0b0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.317+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c10f5f0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.318+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c05f5f0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.319+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7efd4c007c90 con 0x7efd5c0fcef0 2024-04-16T07:24:52.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.320+0000 7efd52a00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 0x7efd40071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.320+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd4c08e3a0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.320+0000 7efd5a000700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 0x7efd40071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.321+0000 7efd5a000700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 0x7efd40071150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd48009fd0 tx=0x7efd48009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:52.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.325+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efd4c05abd0 con 0x7efd5c0fcef0 2024-04-16T07:24:52.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 15) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:52.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.327+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:52.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.328+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 19 ==== client_session(renewcaps seq 15) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:52.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:52.328+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 15) v1 from mds.0 2024-04-16T07:24:52.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.586+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd5c10fa20 con 0x7efd4006eca0 2024-04-16T07:24:52.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.588+0000 7efd52a00700 1 -- 172.21.15.26:0/3155315195 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7efd5c10fa20 con 0x7efd4006eca0 2024-04-16T07:24:52.587 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr" 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "", 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:24:52.588 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.591+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 msgr2=0x7efd40071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.591+0000 7efd61e00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 0x7efd40071150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd48009fd0 tx=0x7efd48009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.591+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 msgr2=0x7efd5c10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.591+0000 7efd61e00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd4c001ce0 tx=0x7efd4c009e60 comp rx=0 tx=0).stop 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 shutdown_connections 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efd4006eca0 0x7efd40071150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd5c0fcef0 0x7efd5c10da90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 --2- 172.21.15.26:0/3155315195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd5c0fe0e0 0x7efd5c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 >> 172.21.15.26:0/3155315195 conn(0x7efd5c0f8530 msgr2=0x7efd5c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 shutdown_connections 2024-04-16T07:24:52.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.592+0000 7efd61e00700 1 -- 172.21.15.26:0/3155315195 wait complete. 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 -- 172.21.15.26:0/2674171143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fce40 msgr2=0x7fd8b40fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2674171143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fce40 0x7fd8b40fd250 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4004940 tx=0x7fd8a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 -- 172.21.15.26:0/2674171143 shutdown_connections 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2674171143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fe040 0x7fd8b40fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2674171143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fce40 0x7fd8b40fd250 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 -- 172.21.15.26:0/2674171143 >> 172.21.15.26:0/2674171143 conn(0x7fd8b40f8530 msgr2=0x7fd8b40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 -- 172.21.15.26:0/2674171143 shutdown_connections 2024-04-16T07:24:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.736+0000 7fd8b9600700 1 -- 172.21.15.26:0/2674171143 wait complete. 2024-04-16T07:24:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 Processor -- start 2024-04-16T07:24:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 -- start start 2024-04-16T07:24:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fce40 0x7fd8b410fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4110f90 con 0x7fd8b40fe040 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.737+0000 7fd8b9600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4111100 con 0x7fd8b40fce40 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56272/0 (socket says 172.21.15.26:56272) 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 -- 172.21.15.26:0/2193633328 learned_addr learned my addr 172.21.15.26:0/2193633328 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:24:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2a00700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fce40 0x7fd8b410fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 -- 172.21.15.26:0/2193633328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fce40 msgr2=0x7fd8b410fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fce40 0x7fd8b410fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 -- 172.21.15.26:0/2193633328 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a4005040 con 0x7fd8b40fe040 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.738+0000 7fd8b2000700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000b790 tx=0x7fd8a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.739+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a000f6f0 con 0x7fd8b40fe040 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.739+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8a000fd20 con 0x7fd8b40fe040 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.739+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b41113e0 con 0x7fd8b40fe040 2024-04-16T07:24:52.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.739+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a00103a0 con 0x7fd8b40fe040 2024-04-16T07:24:52.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.739+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b41119a0 con 0x7fd8b40fe040 2024-04-16T07:24:52.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.740+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b41019d0 con 0x7fd8b40fe040 2024-04-16T07:24:52.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.741+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fd8a0011400 con 0x7fd8b40fe040 2024-04-16T07:24:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.742+0000 7fd8aaa00700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 0x7fd898071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:24:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.742+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd8a008e7b0 con 0x7fd8b40fe040 2024-04-16T07:24:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.742+0000 7fd8b2a00700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 0x7fd898071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:24:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.743+0000 7fd8b2a00700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 0x7fd898071150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b3b0 tx=0x7fd8a4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:24:52.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:52.747+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd8a005b040 con 0x7fd8b40fe040 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:53.038+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:53.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.066+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd8b405f5f0 con 0x7fd8b40fe040 2024-04-16T07:24:53.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.066+0000 7fd8aaa00700 1 -- 172.21.15.26:0/2193633328 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fd8a005abd0 con 0x7fd8b40fe040 2024-04-16T07:24:53.065 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 msgr2=0x7fd898071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 0x7fd898071150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b3b0 tx=0x7fd8a4009e30 comp rx=0 tx=0).stop 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 msgr2=0x7fd8b4110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000b790 tx=0x7fd8a0009250 comp rx=0 tx=0).stop 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 shutdown_connections 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.069+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd89806eca0 0x7fd898071150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.070+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8b40fce40 0x7fd8b410fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.070+0000 7fd8b9600700 1 --2- 172.21.15.26:0/2193633328 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b40fe040 0x7fd8b4110280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.070+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 >> 172.21.15.26:0/2193633328 conn(0x7fd8b40f8530 msgr2=0x7fd8b40f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.070+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 shutdown_connections 2024-04-16T07:24:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:24:53.070+0000 7fd8b9600700 1 -- 172.21.15.26:0/2193633328 wait complete. 2024-04-16T07:24:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:52 smithi104 ceph-mon[36163]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:52 smithi104 ceph-mon[36163]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:52 smithi104 ceph-mon[36163]: mgrmap e35: smithi026.tawexg(active, since 2s), standbys: smithi104.uqfljg 2024-04-16T07:24:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:52 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2423791853' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:24:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:52 smithi026 ceph-mon[92215]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:52 smithi026 ceph-mon[92215]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:52 smithi026 ceph-mon[92215]: mgrmap e35: smithi026.tawexg(active, since 2s), standbys: smithi104.uqfljg 2024-04-16T07:24:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:52 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2423791853' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:53.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:53.327+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:54.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:54.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:54.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:54.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:54.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:54.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:54.039+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:54.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:53 smithi026 ceph-mon[92215]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:54.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:53 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2193633328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:24:54.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:53 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:54.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:54.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:54.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:54.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:54.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:54.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:54.327+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:54.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:53 smithi104 ceph-mon[36163]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:24:54.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:53 smithi104 ceph-mon[36163]: from='client.? 172.21.15.26:0/2193633328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:24:54.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:53 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:54 smithi026 ceph-mon[92215]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:54 smithi026 ceph-mon[92215]: mgrmap e36: smithi026.tawexg(active, since 4s), standbys: smithi104.uqfljg 2024-04-16T07:24:55.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:55.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:55.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:55.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:55.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:55.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:55.038+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:55.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:55.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:55.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:55.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:55.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:55.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:55.328+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:54 smithi104 ceph-mon[36163]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:54 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:54 smithi104 ceph-mon[36163]: mgrmap e36: smithi026.tawexg(active, since 4s), standbys: smithi104.uqfljg 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.038+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.038+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.038+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.039+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.039+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:56.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:56.039+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:56.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:56.328+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:57.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:57.039+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:56 smithi026 ceph-mon[92215]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:56 smithi104 ceph-mon[36163]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:24:57.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:56 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:56 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:56 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:56 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:57.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:57.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:57.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:57.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:57.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:57.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:57.328+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:58.039+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:58.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:58.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:58.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:58.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:58.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:58.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:58.328+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:59.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:24:59.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:24:59.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:24:59.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:24:59.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:24:59.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:24:59.040+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:59.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:58 smithi026 ceph-mon[92215]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:24:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:24:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:24:59.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:24:59.329+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:24:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:58 smithi104 ceph-mon[36163]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T07:24:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:58 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:24:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:24:58 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:00.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77121 2024-04-16T07:25:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 180, success, outsize: 120 2024-04-16T07:25:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.012+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.012+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.012+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.012+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.012+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77210 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 182, success, outsize: 120 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.037+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.037+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.037+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.037+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.037+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:00.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:00.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:00.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:00.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:00.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:00.039+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:00.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77302 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.055+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 184, success, outsize: 120 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.055+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.055+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.055+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.055+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77400 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 186, success, outsize: 120 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.076+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.076+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.076+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.076+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.077+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77495 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 188, success, outsize: 120 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.095+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.095+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.095+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.095+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.095+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77584 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 190, success, outsize: 120 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.112+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.112+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.112+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.112+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.112+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77676 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 192, success, outsize: 120 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.130+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.130+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.130+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.130+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.130+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77774 2024-04-16T07:25:00.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 194, success, outsize: 120 2024-04-16T07:25:00.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.147+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.147+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.147+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.147+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.147+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77869 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 196, success, outsize: 120 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.164+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.164+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.164+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.164+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.164+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77958 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 198, success, outsize: 120 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.180+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.180+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.180+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.180+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.180+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78050 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 200, success, outsize: 120 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.197+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.197+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.197+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.197+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.197+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78148 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 202, success, outsize: 120 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.216+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.216+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.216+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.216+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.216+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78243 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 204, success, outsize: 120 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.235+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.235+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.235+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.235+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.235+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78332 2024-04-16T07:25:00.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 206, success, outsize: 120 2024-04-16T07:25:00.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.251+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.251+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.251+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.251+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.251+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78424 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 208, success, outsize: 120 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.268+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.268+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.268+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.268+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.268+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78522 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 210, success, outsize: 120 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.286+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.286+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.286+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.286+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.286+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78617 2024-04-16T07:25:00.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.304+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 212, success, outsize: 120 2024-04-16T07:25:00.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.304+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.304+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.304+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.304+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78706 2024-04-16T07:25:00.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 214, success, outsize: 120 2024-04-16T07:25:00.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.319+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.319+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.319+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.319+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.319+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:00.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:00.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:00.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 105 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:00.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:00.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.329+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78798 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 216, success, outsize: 120 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.334+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.334+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.334+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.334+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.334+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78896 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 218, success, outsize: 120 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.351+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.351+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.351+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.351+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.351+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78991 2024-04-16T07:25:00.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 220, success, outsize: 120 2024-04-16T07:25:00.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.369+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.369+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.369+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.369+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.369+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79080 2024-04-16T07:25:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 222, success, outsize: 120 2024-04-16T07:25:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.388+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.388+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.388+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.388+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.388+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79172 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 224, success, outsize: 120 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.407+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.407+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.407+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.407+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.407+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79270 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 226, success, outsize: 120 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.427+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.427+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.427+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.427+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.427+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79365 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 228, success, outsize: 120 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.442+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.442+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.442+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.442+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.442+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79454 2024-04-16T07:25:00.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 230, success, outsize: 120 2024-04-16T07:25:00.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.457+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.457+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.457+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.457+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.457+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79547 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 232, success, outsize: 120 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.477+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.477+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.477+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.477+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.477+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:00.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79646 2024-04-16T07:25:00.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 234, success, outsize: 120 2024-04-16T07:25:00.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.491+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:00.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.491+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:00.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.491+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:00.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.491+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:00.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:00.492+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:01.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:01.040+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:01.259 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:00 smithi104 ceph-mon[36163]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:25:01.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:00 smithi026 ceph-mon[92215]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:01.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:01.329+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:02.040+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:02.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:02.329+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:25:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:02.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:25:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:25:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:02 smithi026 ceph-mon[92215]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-16T07:25:02.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:02 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:03.040+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:03.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.329+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:03.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:03.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.329+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:03.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.329+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:03.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.330+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:03.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:03.330+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:03.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:03 smithi026 ceph-mon[92215]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:25:03.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:03 smithi026 ceph-mon[92215]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:25:03.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:03 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:25:03.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:03 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T07:25:04.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:04.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 16) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.040+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:04.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.041+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 21 ==== client_session(renewcaps seq 16) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:25:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:04.041+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 16) v1 from mds.0 2024-04-16T07:25:04.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:04.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:04.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:04.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:04.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:04.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:04.330+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:25:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: Updating smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: Updating smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T07:25:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:04 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:05.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:05.041+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:05.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:05.330+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:06.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:06.041+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:06.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:06.072 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:25:06.072 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:06.072 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:05 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:06.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:06.330+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:07.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:07.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:07.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:07.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:07.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:07.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:07.041+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:07.069 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:06 smithi026 ceph-mon[92215]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:06 smithi104 ceph-mon[36163]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:07.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:07.330+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.041+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.041+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.041+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.041+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.042+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:08.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:08.042+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:08.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:08.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:08.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:08.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:08.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:08.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:08.331+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: Upgrade: Updating mon.smithi104 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:08.654 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:08 smithi104 ceph-mon[36163]: Deploying daemon mon.smithi104 on smithi104 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: Upgrade: Updating mon.smithi104 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:08.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:08 smithi026 ceph-mon[92215]: Deploying daemon mon.smithi104 on smithi104 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:09.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:09.042+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:09.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:09.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:09.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:09.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:09.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:09.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:09.331+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:10.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:10.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:10.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:10.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:10.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:10.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.042+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.276+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 msgr2=0x55fc3f5143c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-04-16T07:25:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.276+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 msgr2=0x55fc3f5143c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:25:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.276+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800afa0 tx=0x7fe9c8004b80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:25:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.276+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800afa0 tx=0x7fe9c8004b80 comp rx=0 tx=0).stop 2024-04-16T07:25:10.276 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:09 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:25:10.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.277+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 msgr2=0x55fc3f5143c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:25:10.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.277+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f500090 0x55fc3f5143c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:10.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.277+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b40085b0 0x7fe9b400a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:10.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.278+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:10.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.278+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b40085b0 msgr2=0x7fe9b400a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:25:10.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.279+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b40085b0 0x7fe9b400a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:25:10.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.279+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400da90 con 0x55fc3f5009a0 2024-04-16T07:25:10.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.279+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400dc00 con 0x7fe9b40085b0 2024-04-16T07:25:10.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.279+0000 7fe9baa00640 0 client.24339 ms_handle_reset on v2:172.21.15.104:3300/0 2024-04-16T07:25:10.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.279+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:25:10.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 msgr2=0x7fbf7000cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-04-16T07:25:10.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 msgr2=0x7fbf7000cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00a5c0 tx=0x7fbf8c005af0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00a5c0 tx=0x7fbf8c005af0 comp rx=0 tx=0).stop 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 msgr2=0x7fbf7000cfc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa01090 0x7fbf7000cfc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa15bb0 0x7fbf7000fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.277+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.278+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000ab00 con 0x7fbf70008390 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.278+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70010ab0 con 0x563ccaa15bb0 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.278+0000 7fbf77400640 0 client.14524 ms_handle_reset on v2:172.21.15.104:3300/0 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.279+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa15bb0 msgr2=0x7fbf7000fd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.279+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa15bb0 0x7fbf7000fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:25:10.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.279+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:25:10.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.282+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:10.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.282+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:10.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.284+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b40085b0 msgr2=0x7fe9b400a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:25:10.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.284+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b40085b0 0x7fe9b400a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:10.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.284+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,mdsmap=16+,monmap=3+}) v3 -- 0x7fe9a0001d80 con 0x55fc3f5009a0 2024-04-16T07:25:10.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.285+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa15bb0 msgr2=0x7fbf7000fd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:25:10.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.285+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563ccaa15bb0 0x7fbf7000fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:10.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.286+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,mdsmap=16+,monmap=3+}) v3 -- 0x7fbf70010bf0 con 0x7fbf70008390 2024-04-16T07:25:10.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.286+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000a5c0 tx=0x7fe9d0005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:10.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.286+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-16T07:25:10.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.288+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005930 tx=0x7fbf8c005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:10.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.288+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-16T07:25:10.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.289+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c010920 con 0x7fbf70008390 2024-04-16T07:25:10.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.291+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d00105e0 con 0x55fc3f5009a0 2024-04-16T07:25:10.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:10.292+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf8c010a80 con 0x7fbf70008390 2024-04-16T07:25:10.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.297+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d0010c10 con 0x55fc3f5009a0 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:10.331+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:10.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:10 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104[36151]: 2024-04-16T07:25:10.275+0000 7f46ca600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:25:10.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:10 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104[36151]: 2024-04-16T07:25:10.275+0000 7f46ca600700 -1 mon.smithi104@1(peon) e2 *** Got Signal Terminated *** 2024-04-16T07:25:10.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:10 smithi104 podman[80867]: 2024-04-16 07:25:10.451328244 +0000 UTC m=+0.312417049 container died dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:25:11.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:11.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:11.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:11.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:11.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:11.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:11.042+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:11.278 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:10 smithi104 podman[80867]: 2024-04-16 07:25:10.986211725 +0000 UTC m=+0.847300560 container cleanup dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-16T07:25:11.278 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:10 smithi104 bash[80867]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:11.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:11.331+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:11.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:11 smithi104 podman[80881]: 2024-04-16 07:25:11.278963374 +0000 UTC m=+0.814573181 container remove dd8dd9b5b0a8bdc0925e1a1a9635d966f07b2fd4deb2eff30e5567d8610744e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:12.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:12.043+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:11 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi104.service: Deactivated successfully. 2024-04-16T07:25:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:11 smithi104 systemd[1]: Stopped Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:25:12.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:11 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi104.service: Consumed 14.013s CPU time. 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 16) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:12.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:12.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:25:12.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:12.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.331+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:12.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.332+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 20 ==== client_session(renewcaps seq 16) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:12.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:12.332+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 16) v1 from mds.0 2024-04-16T07:25:12.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 systemd[1]: Starting Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:25:12.817 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 podman[81008]: 2024-04-16 07:25:12.411902252 +0000 UTC m=+0.035253862 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:25:12.817 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 podman[81008]: 2024-04-16 07:25:12.512987337 +0000 UTC m=+0.136338978 container create 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 podman[81008]: 2024-04-16 07:25:12.817448337 +0000 UTC m=+0.440800031 container init 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 podman[81008]: 2024-04-16 07:25:12.829125259 +0000 UTC m=+0.452476936 container start 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: pidfile_write: ignore empty --pid-file 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: load: jerasure load: lrc 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: RocksDB version: 7.9.2 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Git sha 0 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: DB SUMMARY 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: DB Session ID: K3KFM4P96LLL054BUM6G 2024-04-16T07:25:12.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: CURRENT file: CURRENT 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 1, files: 000027.sst 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 000025.log size: 4759311 ; 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.error_if_exists: 0 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.create_if_missing: 0 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.paranoid_checks: 1 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.env: 0x5586919c2cc0 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.info_log: 0x5586a6091ae0 2024-04-16T07:25:12.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.statistics: (nil) 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.use_fsync: 0 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_log_file_size: 0 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_fallocate: 1 2024-04-16T07:25:12.897 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.use_direct_reads: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.db_log_dir: 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.wal_dir: 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T07:25:12.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.write_buffer_manager: 0x5586a60621e0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.unordered_write: 0 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T07:25:12.899 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.row_cache: None 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.wal_filter: None 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.two_write_queues: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.wal_compression: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.atomic_flush: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.log_readahead_size: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T07:25:12.900 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_background_jobs: 2 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_background_compactions: -1 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_subcompactions: 1 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T07:25:12.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_open_files: -1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_background_flushes: -1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Compression algorithms supported: 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kZSTD supported: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kXpressCompression supported: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kZlibCompression supported: 1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kSnappyCompression supported: 1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kLZ4Compression supported: 1 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: kBZip2Compression supported: 0 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T07:25:12.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000009 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.merge_operator: 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_filter: None 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586a6091cc0) 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-16T07:25:12.903 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_type: 0 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_index_type: 0 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_shortening: 1 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: checksum: 4 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: no_block_cache: 0 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache: 0x5586a6095090 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_options: 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: capacity : 536870912 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_shard_bits : 4 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: strict_capacity_limit : 0 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_compressed: (nil) 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: persistent_cache: (nil) 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size: 4096 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size_deviation: 10 2024-04-16T07:25:12.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_restart_interval: 16 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_block_restart_interval: 1 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_block_size: 4096 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: partition_filters: 0 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: use_delta_encoding: 1 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: filter_policy: bloomfilter 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: whole_key_filtering: 1 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: verify_compression: 0 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: format_version: 5 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_index_compression: 1 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_align: 0 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_auto_readahead_size: 262144 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: prepopulate_block_cache: 0 2024-04-16T07:25:12.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: initial_auto_readahead_size: 8192 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression: NoCompression 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T07:25:12.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.num_levels: 7 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T07:25:12.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T07:25:12.908 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T07:25:12.909 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.inplace_update_support: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.bloom_locality: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.max_successive_merges: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T07:25:12.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.ttl: 2592000 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enable_blob_files: false 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.min_blob_size: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T07:25:12.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 11932, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9adcd018-f940-4d6a-9eae-b81dad9afac0 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252312885258, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-16T07:25:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:13.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.043+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252312905501, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2962263, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11937, "largest_seqno": 13204, "table_properties": {"data_size": 2956292, "index_size": 3582, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12488, "raw_average_key_size": 23, "raw_value_size": 2944747, "raw_average_value_size": 5535, "num_data_blocks": 164, "num_entries": 532, "num_filter_entries": 532, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713252312, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9adcd018-f940-4d6a-9eae-b81dad9afac0", "db_session_id": "K3KFM4P96LLL054BUM6G", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-16T07:25:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713252312905682, "job": 1, "event": "recovery_finished"} 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi104/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5586a6184000 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: DB pointer 0x5586a616e000 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** DB Stats ** 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:25:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 1/0 2.83 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-16T07:25:13.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L6 1/0 5.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 2/0 8.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 407.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T07:25:13.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 107.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 107.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Block cache BinnedLRUCache@0x5586a6095090#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: starting mon.smithi104 rank 1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi104 fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???) e2 preinit fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).mds e16 new map 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).mds e16 print_map 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: e16 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: legacy client fscid: 1 2024-04-16T07:25:13.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Filesystem 'cephfs' (1) 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: fs_name cephfs 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: epoch 15 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: tableserver 0 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: root 0 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: session_timeout 60 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: session_autoclose 300 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_file_size 1099511627776 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_xattr_size 65536 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: required_client_features {} 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: last_failure 0 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: last_failure_osd_epoch 47 2024-04-16T07:25:13.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_mds 1 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: in 0 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: up {0=14470} 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: failed 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: damaged 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: stopped 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_pools [3] 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_pool 2 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: inline_data disabled 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: balancer 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: bal_rank_mask -1 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: standby_count_wanted 1 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: qdb_cluster leader: 0 members: 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Standby daemons: 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 ceph-mon[81027]: mon.smithi104@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 bash[81008]: 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 2024-04-16T07:25:13.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:12 smithi104 systemd[1]: Started Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:25:13.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:13.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:13.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:13.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:13.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:13.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.332+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:13.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:13.502+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d000fdd0 con 0x55fc3f5009a0 2024-04-16T07:25:13.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:13.501+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c00f950 con 0x7fbf70008390 2024-04-16T07:25:13.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:13 smithi026 ceph-mon[92215]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:13.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:13 smithi104 ceph-mon[81027]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:14.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:14.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:14.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:14.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:14.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:14.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:14.043+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:14.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:14.332+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: pgmap v15: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:25:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: mon.smithi026 calling monitor election 2024-04-16T07:25:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: mon.smithi104 calling monitor election 2024-04-16T07:25:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: monmap epoch 3 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: last_changed 2024-04-16T07:25:13.493441+0000 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: created 2024-04-16T07:10:43.696046+0000 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: min_mon_release 19 (squid) 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: election_strategy: 1 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.smithi026 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: mgrmap e36: smithi026.tawexg(active, since 24s), standbys: smithi104.uqfljg 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: overall HEALTH_OK 2024-04-16T07:25:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: pgmap v15: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: mon.smithi026 calling monitor election 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: mon.smithi104 calling monitor election 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-16T07:25:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: mon.smithi026 is new leader, mons smithi026,smithi104 in quorum (ranks 0,1) 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: monmap epoch 3 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: last_changed 2024-04-16T07:25:13.493441+0000 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: created 2024-04-16T07:10:43.696046+0000 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: min_mon_release 19 (squid) 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: election_strategy: 1 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.smithi026 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: mgrmap e36: smithi026.tawexg(active, since 24s), standbys: smithi104.uqfljg 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: overall HEALTH_OK 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:25:15.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:15.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:15.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:15.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:15.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:15.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:15.043+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:15.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:15.332+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:16.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:16.044+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:16.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:16.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:16.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:16.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:16.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:16.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:16.332+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:16 smithi026 ceph-mon[92215]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:16.791 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:16 smithi104 ceph-mon[81027]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:17.044+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:17.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:17.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:17.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:17.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:17.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:17.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:17.333+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:18.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:18.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:18.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:18.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:18.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:18.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:18.044+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:18.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:18.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:18.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:18.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:18.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:18.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:18.333+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:19.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:18 smithi026 ceph-mon[92215]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:19.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:18 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:19.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:18 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:19.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.044+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:19.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.044+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:19.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.044+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:19.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.044+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:19.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.045+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:19.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:19.045+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:18 smithi104 ceph-mon[81027]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:18 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:18 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:19.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:19.333+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:20.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:20.045+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:20.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:20.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:20.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:20.333+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:21.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:21.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:21.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:21.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:21.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:21.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:21.045+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:21.160 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:20 smithi104 ceph-mon[81027]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:21.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:21.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:21.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:20 smithi026 ceph-mon[92215]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:21.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:21.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:21.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:21.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:21.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:21.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:21.333+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:22.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:22.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:22.334+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.045+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:23.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:23.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:23.243 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:22 smithi026 ceph-mon[92215]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:23.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 -- 172.21.15.26:0/3591908860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4073170 msgr2=0x7fe3d4073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 --2- 172.21.15.26:0/3591908860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4073170 0x7fe3d4073b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0004940 tx=0x7fe3c0009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:23.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 -- 172.21.15.26:0/3591908860 shutdown_connections 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 --2- 172.21.15.26:0/3591908860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4073170 0x7fe3d4073b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 --2- 172.21.15.26:0/3591908860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d4072820 0x7fe3d4072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 -- 172.21.15.26:0/3591908860 >> 172.21.15.26:0/3591908860 conn(0x7fe3d40fa950 msgr2=0x7fe3d40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 -- 172.21.15.26:0/3591908860 shutdown_connections 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.243+0000 7fe3da800700 1 -- 172.21.15.26:0/3591908860 wait complete. 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.244+0000 7fe3da800700 1 Processor -- start 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.244+0000 7fe3da800700 1 -- start start 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3da800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3da800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 0x7fe3d41153f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3da800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d4114ab0 con 0x7fe3d4072820 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3da800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d4114bf0 con 0x7fe3d41145f0 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3d3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3d3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42066/0 (socket says 172.21.15.26:42066) 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3d3400700 1 -- 172.21.15.26:0/4127168435 learned_addr learned my addr 172.21.15.26:0/4127168435 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.245+0000 7fe3d2a00700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 0x7fe3d41153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3d3400700 1 -- 172.21.15.26:0/4127168435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 msgr2=0x7fe3d41153f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3d3400700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 0x7fe3d41153f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3d3400700 1 -- 172.21.15.26:0/4127168435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0005040 con 0x7fe3d4072820 2024-04-16T07:25:23.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3d2a00700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 0x7fe3d41153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3d3400700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4002a50 tx=0x7fe3c4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c4007cc0 con 0x7fe3d4072820 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3c400f040 con 0x7fe3d4072820 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c400da40 con 0x7fe3d4072820 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d4115a50 con 0x7fe3d4072820 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.246+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d4115fe0 con 0x7fe3d4072820 2024-04-16T07:25:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.248+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d410c720 con 0x7fe3d4072820 2024-04-16T07:25:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.249+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe3c40097c0 con 0x7fe3d4072820 2024-04-16T07:25:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.250+0000 7fe3cb400700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 0x7fe3b80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.250+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3c408e820 con 0x7fe3d4072820 2024-04-16T07:25:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.250+0000 7fe3d2a00700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 0x7fe3b80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.251+0000 7fe3d2a00700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 0x7fe3b80711d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000b7a0 tx=0x7fe3c0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:23.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.256+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe3c405ac50 con 0x7fe3d4072820 2024-04-16T07:25:23.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:22 smithi104 ceph-mon[81027]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:23.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:23.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:23.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:23.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 115 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:23.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:23.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.334+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:23.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.507+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3d4073170 con 0x7fe3b806ed20 2024-04-16T07:25:23.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.510+0000 7fe3cb400700 1 -- 172.21.15.26:0/4127168435 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fe3d4073170 con 0x7fe3b806ed20 2024-04-16T07:25:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.512+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 msgr2=0x7fe3b80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.512+0000 7fe3da800700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 0x7fe3b80711d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000b7a0 tx=0x7fe3c0009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.512+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 msgr2=0x7fe3d41140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.512+0000 7fe3da800700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4002a50 tx=0x7fe3c4004600 comp rx=0 tx=0).stop 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 shutdown_connections 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe3b806ed20 0x7fe3b80711d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3d4072820 0x7fe3d41140b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 --2- 172.21.15.26:0/4127168435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3d41145f0 0x7fe3d41153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 >> 172.21.15.26:0/4127168435 conn(0x7fe3d40fa950 msgr2=0x7fe3d40fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 shutdown_connections 2024-04-16T07:25:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.513+0000 7fe3da800700 1 -- 172.21.15.26:0/4127168435 wait complete. 2024-04-16T07:25:23.528 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:25:23.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 81920 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 236, success, outsize: 120 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.677+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.677+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.677+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.677+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.677+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.677+0000 7f54e9800700 1 -- 172.21.15.26:0/3197553845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e40ff020 msgr2=0x7f54e40ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.677+0000 7f54e9800700 1 --2- 172.21.15.26:0/3197553845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e40ff020 0x7f54e40ff430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f54d4004940 tx=0x7f54d4009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 -- 172.21.15.26:0/3197553845 shutdown_connections 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 --2- 172.21.15.26:0/3197553845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e41002d0 0x7f54e4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 --2- 172.21.15.26:0/3197553845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e40ff020 0x7f54e40ff430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 -- 172.21.15.26:0/3197553845 >> 172.21.15.26:0/3197553845 conn(0x7f54e40fa950 msgr2=0x7f54e40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 -- 172.21.15.26:0/3197553845 shutdown_connections 2024-04-16T07:25:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.678+0000 7f54e9800700 1 -- 172.21.15.26:0/3197553845 wait complete. 2024-04-16T07:25:23.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 Processor -- start 2024-04-16T07:25:23.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 -- start start 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 0x7f54e4110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e4110620 con 0x7f54e41002d0 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.679+0000 7f54e9800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e4110790 con 0x7f54e40ff020 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 0x7f54e4110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.26:43610/0 (socket says 172.21.15.26:43610) 2024-04-16T07:25:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2a00700 1 -- 172.21.15.26:0/4290575878 learned_addr learned my addr 172.21.15.26:0/4290575878 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2000700 1 -- 172.21.15.26:0/4290575878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 msgr2=0x7f54e410fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2000700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2000700 1 -- 172.21.15.26:0/4290575878 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4005040 con 0x7f54e41002d0 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.680+0000 7f54e2a00700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54e2000700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 0x7f54e4110020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009fd0 tx=0x7f54d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d0004e90 con 0x7f54e41002d0 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54d000f040 con 0x7f54e41002d0 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e4115190 con 0x7f54e41002d0 2024-04-16T07:25:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d000ca50 con 0x7f54e41002d0 2024-04-16T07:25:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.681+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e4115750 con 0x7f54e41002d0 2024-04-16T07:25:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.683+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e404c710 con 0x7f54e41002d0 2024-04-16T07:25:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.684+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f54d0007cc0 con 0x7f54e41002d0 2024-04-16T07:25:23.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.685+0000 7f54daa00700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 0x7f54c80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:23.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.685+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f54d008e520 con 0x7f54e41002d0 2024-04-16T07:25:23.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.685+0000 7f54e2a00700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 0x7f54c80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.686+0000 7f54e2a00700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 0x7f54c80711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009fd0 tx=0x7f54d4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:23.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.691+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f54d005ac90 con 0x7f54e41002d0 2024-04-16T07:25:23.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 81998 2024-04-16T07:25:23.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 238, success, outsize: 120 2024-04-16T07:25:23.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.695+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.695+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.695+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.695+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.695+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82079 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 240, success, outsize: 120 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.709+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.710+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.710+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.710+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.710+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82166 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 242, success, outsize: 120 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.725+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.725+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.725+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.725+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.725+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82250 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 244, success, outsize: 120 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.741+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.741+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.741+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.741+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.742+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82328 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 246, success, outsize: 120 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.757+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.757+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.757+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.757+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.757+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82409 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 248, success, outsize: 120 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.772+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.772+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.772+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.772+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.772+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82496 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 250, success, outsize: 120 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.790+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.790+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.790+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.790+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.790+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82580 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 252, success, outsize: 120 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.808+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.808+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.808+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.808+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.808+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82658 2024-04-16T07:25:23.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 254, success, outsize: 120 2024-04-16T07:25:23.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.825+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.825+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.825+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.825+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.825+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82739 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 256, success, outsize: 120 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.843+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.843+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.843+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.843+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.843+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82826 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 258, success, outsize: 120 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.863+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.863+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.863+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.863+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.863+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82910 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 260, success, outsize: 120 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.885+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.885+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.885+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.885+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.885+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82988 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 262, success, outsize: 120 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.903+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.903+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.903+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.903+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.903+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83069 2024-04-16T07:25:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 264, success, outsize: 120 2024-04-16T07:25:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.922+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.922+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.922+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.922+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.922+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83156 2024-04-16T07:25:23.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 266, success, outsize: 120 2024-04-16T07:25:23.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.943+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.943+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.943+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.943+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.943+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.944+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54e41045a0 con 0x7f54c806ed30 2024-04-16T07:25:23.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.947+0000 7f54daa00700 1 -- 172.21.15.26:0/4290575878 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f54e41045a0 con 0x7f54c806ed30 2024-04-16T07:25:23.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 msgr2=0x7f54c80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 0x7f54c80711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009fd0 tx=0x7f54d4009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:23.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 msgr2=0x7f54e4110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 0x7f54e4110020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009fd0 tx=0x7f54d000b040 comp rx=0 tx=0).stop 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 shutdown_connections 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f54c806ed30 0x7f54c80711e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54e40ff020 0x7f54e410fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 --2- 172.21.15.26:0/4290575878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e41002d0 0x7f54e4110020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.950+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 >> 172.21.15.26:0/4290575878 conn(0x7f54e40fa950 msgr2=0x7f54e40fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.951+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 shutdown_connections 2024-04-16T07:25:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:23.951+0000 7f54e9800700 1 -- 172.21.15.26:0/4290575878 wait complete. 2024-04-16T07:25:23.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83240 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 268, success, outsize: 120 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.964+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.964+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.964+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.964+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.964+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:23.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83318 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 270, success, outsize: 120 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.984+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.984+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.984+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.984+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:23.984+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83399 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.005+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 272, success, outsize: 120 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.005+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.005+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.005+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.005+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83486 2024-04-16T07:25:24.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 274, success, outsize: 120 2024-04-16T07:25:24.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.027+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.027+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.027+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.027+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.027+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 17) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:24.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:24.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.047+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 22 ==== client_session(renewcaps seq 17) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:25:24.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:24.047+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 17) v1 from mds.0 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83570 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 276, success, outsize: 120 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.049+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.049+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.049+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.049+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.049+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83648 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 278, success, outsize: 120 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.070+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.070+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.070+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.070+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.070+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83729 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.091+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 280, success, outsize: 120 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.091+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.091+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.091+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.091+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.103+0000 7f4840c00700 1 -- 172.21.15.26:0/2865558284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 msgr2=0x7f483c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.103+0000 7f4840c00700 1 --2- 172.21.15.26:0/2865558284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c102390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4828004940 tx=0x7f4828009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 -- 172.21.15.26:0/2865558284 shutdown_connections 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 --2- 172.21.15.26:0/2865558284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c102390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 --2- 172.21.15.26:0/2865558284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 0x7f483c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 -- 172.21.15.26:0/2865558284 >> 172.21.15.26:0/2865558284 conn(0x7f483c0f74c0 msgr2=0x7f483c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 -- 172.21.15.26:0/2865558284 shutdown_connections 2024-04-16T07:25:24.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.104+0000 7f4840c00700 1 -- 172.21.15.26:0/2865558284 wait complete. 2024-04-16T07:25:24.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 Processor -- start 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 -- start start 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 0x7f483c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c10e510 con 0x7f483c0fff70 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.105+0000 7f4840c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c10ed60 con 0x7f483c0662f0 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42128/0 (socket says 172.21.15.26:42128) 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 -- 172.21.15.26:0/2445571524 learned_addr learned my addr 172.21.15.26:0/2445571524 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839e00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 0x7f483c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 -- 172.21.15.26:0/2445571524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 msgr2=0x7f483c10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 0x7f483c10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 -- 172.21.15.26:0/2445571524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828005040 con 0x7f483c0fff70 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.106+0000 7f4839400700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f482800b0d0 tx=0x7f4828009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.107+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482800e070 con 0x7f483c0fff70 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.107+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f482800b410 con 0x7f483c0fff70 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.107+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c10efe0 con 0x7f483c0fff70 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.107+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48280125e0 con 0x7f483c0fff70 2024-04-16T07:25:24.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.107+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c10f520 con 0x7f483c0fff70 2024-04-16T07:25:24.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.108+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c05f5f0 con 0x7f483c0fff70 2024-04-16T07:25:24.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.109+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f482800b580 con 0x7f483c0fff70 2024-04-16T07:25:24.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.110+0000 7f4832000700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 0x7f48200711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.110+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f482808f7b0 con 0x7f483c0fff70 2024-04-16T07:25:24.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.110+0000 7f4839e00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 0x7f48200711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.111+0000 7f4839e00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 0x7f48200711d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f482c009b80 tx=0x7f482c00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83816 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 282, success, outsize: 120 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.114+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.114+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.114+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.114+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.114+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.115+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f482805bef0 con 0x7f483c0fff70 2024-04-16T07:25:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83900 2024-04-16T07:25:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.138+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 284, success, outsize: 120 2024-04-16T07:25:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.138+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.138+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.138+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.138+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83978 2024-04-16T07:25:24.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 286, success, outsize: 120 2024-04-16T07:25:24.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.160+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.160+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.160+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.160+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.160+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84059 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 288, success, outsize: 120 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.182+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.182+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.182+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.182+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.183+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84146 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 290, success, outsize: 120 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.206+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.206+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.206+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.206+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:25:24.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.206+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:24.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:24.334+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.357+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f483c10f8c0 con 0x7f482006ed20 2024-04-16T07:25:24.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.388+0000 7f4832000700 1 -- 172.21.15.26:0/2445571524 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f483c10f8c0 con 0x7f482006ed20 2024-04-16T07:25:24.387 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:25:24.387 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (2m) 27s ago 12m 14.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:25:24.387 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (13m) 27s ago 13m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:25:24.387 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (11m) 6s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (2m) 27s ago 12m 72.4M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (5m) 27s ago 5m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (5m) 27s ago 5m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (5m) 6s ago 5m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (5m) 6s ago 5m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (115s) 27s ago 14m 479M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (65s) 6s ago 11m 449M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (45s) 27s ago 14m 29.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (11s) 6s ago 11m 20.3M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (2m) 27s ago 12m 7894k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (2m) 6s ago 11m 7696k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (10m) 27s ago 10m 46.2M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (9m) 27s ago 9m 49.1M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (9m) 27s ago 9m 46.6M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (8m) 6s ago 8m 45.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:25:24.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (8m) 6s ago 8m 44.5M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:25:24.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (7m) 6s ago 7m 43.9M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:25:24.389 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (73s) 27s ago 12m 48.2M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:25:24.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 msgr2=0x7f48200711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 0x7f48200711d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f482c009b80 tx=0x7f482c00b040 comp rx=0 tx=0).stop 2024-04-16T07:25:24.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 msgr2=0x7f483c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f482800b0d0 tx=0x7f4828009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 shutdown_connections 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f482006ed20 0x7f48200711d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f483c0662f0 0x7f483c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 --2- 172.21.15.26:0/2445571524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f483c0fff70 0x7f483c10df10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.392+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 >> 172.21.15.26:0/2445571524 conn(0x7f483c0f74c0 msgr2=0x7f483c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.393+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 shutdown_connections 2024-04-16T07:25:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.393+0000 7f4840c00700 1 -- 172.21.15.26:0/2445571524 wait complete. 2024-04-16T07:25:24.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 -- 172.21.15.26:0/3272475405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc1002b0 msgr2=0x7fe5bc100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 --2- 172.21.15.26:0/3272475405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc100720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8004940 tx=0x7fe5a8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 -- 172.21.15.26:0/3272475405 shutdown_connections 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 --2- 172.21.15.26:0/3272475405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc100720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 --2- 172.21.15.26:0/3272475405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 -- 172.21.15.26:0/3272475405 >> 172.21.15.26:0/3272475405 conn(0x7fe5bc0fa790 msgr2=0x7fe5bc0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 -- 172.21.15.26:0/3272475405 shutdown_connections 2024-04-16T07:25:24.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.544+0000 7fe5c1000700 1 -- 172.21.15.26:0/3272475405 wait complete. 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.545+0000 7fe5c1000700 1 Processor -- start 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5c1000700 1 -- start start 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5c1000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5c1000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5c1000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc110670 con 0x7fe5bc0ff000 2024-04-16T07:25:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5c1000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc1107e0 con 0x7fe5bc1002b0 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5b9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5b9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42146/0 (socket says 172.21.15.26:42146) 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.546+0000 7fe5b9e00700 1 -- 172.21.15.26:0/1033174998 learned_addr learned my addr 172.21.15.26:0/1033174998 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9400700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9e00700 1 -- 172.21.15.26:0/1033174998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 msgr2=0x7fe5bc110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9e00700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9e00700 1 -- 172.21.15.26:0/1033174998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a8005040 con 0x7fe5bc0ff000 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9400700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:25:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b9e00700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac002a50 tx=0x7fe5ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:24.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5ac007cc0 con 0x7fe5bc0ff000 2024-04-16T07:25:24.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5ac00f040 con 0x7fe5bc0ff000 2024-04-16T07:25:24.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5ac009640 con 0x7fe5bc0ff000 2024-04-16T07:25:24.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.547+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc115240 con 0x7fe5bc0ff000 2024-04-16T07:25:24.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.548+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc115800 con 0x7fe5bc0ff000 2024-04-16T07:25:24.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.549+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc04c710 con 0x7fe5bc0ff000 2024-04-16T07:25:24.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.550+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe5ac0097c0 con 0x7fe5bc0ff000 2024-04-16T07:25:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.551+0000 7fe5b2000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 0x7fe5a00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.551+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe5ac08e7d0 con 0x7fe5bc0ff000 2024-04-16T07:25:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.551+0000 7fe5b9400700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 0x7fe5a00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.552+0000 7fe5b9400700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 0x7fe5a00711e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800b7a0 tx=0x7fe5a8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:24.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.557+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe5ac05af10 con 0x7fe5bc0ff000 2024-04-16T07:25:24.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.866+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe5bc115c70 con 0x7fe5bc0ff000 2024-04-16T07:25:24.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.867+0000 7fe5b2000700 1 -- 172.21.15.26:0/1033174998 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7fe5ac05aaa0 con 0x7fe5bc0ff000 2024-04-16T07:25:24.866 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:25:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:25:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:25:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:25:24.867 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 msgr2=0x7fe5a00711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 0x7fe5a00711e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800b7a0 tx=0x7fe5a8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 msgr2=0x7fe5bc10fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac002a50 tx=0x7fe5ac0046b0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 shutdown_connections 2024-04-16T07:25:24.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe5a006ed30 0x7fe5a00711e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5bc0ff000 0x7fe5bc10fb30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 --2- 172.21.15.26:0/1033174998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5bc1002b0 0x7fe5bc110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 >> 172.21.15.26:0/1033174998 conn(0x7fe5bc0fa790 msgr2=0x7fe5bc0fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 shutdown_connections 2024-04-16T07:25:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:24.870+0000 7fe5c1000700 1 -- 172.21.15.26:0/1033174998 wait complete. 2024-04-16T07:25:24.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:24 smithi104 ceph-mon[81027]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:24.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:24 smithi104 ceph-mon[81027]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:24.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:24 smithi104 ceph-mon[81027]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:24.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:24 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1033174998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 -- 172.21.15.26:0/3415588992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 msgr2=0x7fbf480fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3415588992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf480fd250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf38004940 tx=0x7fbf38009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 -- 172.21.15.26:0/3415588992 shutdown_connections 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3415588992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf480fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3415588992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf480fd250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 -- 172.21.15.26:0/3415588992 >> 172.21.15.26:0/3415588992 conn(0x7fbf480f8530 msgr2=0x7fbf480fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 -- 172.21.15.26:0/3415588992 shutdown_connections 2024-04-16T07:25:25.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.030+0000 7fbf4e400700 1 -- 172.21.15.26:0/3415588992 wait complete. 2024-04-16T07:25:25.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 Processor -- start 2024-04-16T07:25:25.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 -- start start 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf48070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48070950 con 0x7fbf480fce40 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.031+0000 7fbf4e400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48070ac0 con 0x7fbf480fe040 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42164/0 (socket says 172.21.15.26:42164) 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 -- 172.21.15.26:0/3214206450 learned_addr learned my addr 172.21.15.26:0/3214206450 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf46a00700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf48070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 -- 172.21.15.26:0/3214206450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 msgr2=0x7fbf48070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf48070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 -- 172.21.15.26:0/3214206450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38005040 con 0x7fbf480fce40 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf46a00700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf48070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.032+0000 7fbf47400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800b970 tx=0x7fbf38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.033+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf3800e040 con 0x7fbf480fce40 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.033+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf38004290 con 0x7fbf480fce40 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.033+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf48070d40 con 0x7fbf480fce40 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.033+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf3801c630 con 0x7fbf480fce40 2024-04-16T07:25:25.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.033+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf48115a80 con 0x7fbf480fce40 2024-04-16T07:25:25.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.034+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4805f5f0 con 0x7fbf480fce40 2024-04-16T07:25:25.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.035+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fbf3800bbf0 con 0x7fbf480fce40 2024-04-16T07:25:25.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.036+0000 7fbf3f400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 0x7fbf2c0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.036+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbf3808fe20 con 0x7fbf480fce40 2024-04-16T07:25:25.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.036+0000 7fbf46a00700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 0x7fbf2c0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.037+0000 7fbf46a00700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 0x7fbf2c0711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf34007b00 tx=0x7fbf34008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.041+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbf38058a30 con 0x7fbf480fce40 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:25.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:25.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:25.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:24 smithi026 ceph-mon[92215]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:25.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:24 smithi026 ceph-mon[92215]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:25.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:24 smithi026 ceph-mon[92215]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:25.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:24 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1033174998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.332+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbf48070ed0 con 0x7fbf480fce40 2024-04-16T07:25:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.332+0000 7fbf3f400700 1 -- 172.21.15.26:0/3214206450 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fbf48070ed0 con 0x7fbf480fce40 2024-04-16T07:25:25.332 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:25:25.332 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:25:25.332 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:25:25.333 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:25.335 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:25.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:25.334+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 msgr2=0x7fbf2c0711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 0x7fbf2c0711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf34007b00 tx=0x7fbf34008040 comp rx=0 tx=0).stop 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 msgr2=0x7fbf48071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800b970 tx=0x7fbf38009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 shutdown_connections 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbf2c06ed20 0x7fbf2c0711d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf480fce40 0x7fbf48071e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 --2- 172.21.15.26:0/3214206450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf480fe040 0x7fbf48070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.335+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 >> 172.21.15.26:0/3214206450 conn(0x7fbf480f8530 msgr2=0x7fbf480f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.336+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 shutdown_connections 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.336+0000 7fbf4e400700 1 -- 172.21.15.26:0/3214206450 wait complete. 2024-04-16T07:25:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 -- 172.21.15.26:0/3680933307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 msgr2=0x7f91cc102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 --2- 172.21.15.26:0/3680933307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc102390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91b8004940 tx=0x7f91b8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 -- 172.21.15.26:0/3680933307 shutdown_connections 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 --2- 172.21.15.26:0/3680933307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc102390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 --2- 172.21.15.26:0/3680933307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 0x7f91cc0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.485+0000 7f91d2600700 1 -- 172.21.15.26:0/3680933307 >> 172.21.15.26:0/3680933307 conn(0x7f91cc0f74c0 msgr2=0x7f91cc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.486+0000 7f91d2600700 1 -- 172.21.15.26:0/3680933307 shutdown_connections 2024-04-16T07:25:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.486+0000 7f91d2600700 1 -- 172.21.15.26:0/3680933307 wait complete. 2024-04-16T07:25:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 Processor -- start 2024-04-16T07:25:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 -- start start 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 0x7f91cc10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91cc110850 con 0x7f91cc0fff70 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.487+0000 7f91d2600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91cc110f90 con 0x7f91cc0662f0 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42182/0 (socket says 172.21.15.26:42182) 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 -- 172.21.15.26:0/316427828 learned_addr learned my addr 172.21.15.26:0/316427828 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91cb400700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 0x7f91cc10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 -- 172.21.15.26:0/316427828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 msgr2=0x7f91cc10fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 0x7f91cc10fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.488+0000 7f91caa00700 1 -- 172.21.15.26:0/316427828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8005040 con 0x7f91cc0fff70 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.489+0000 7f91caa00700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009150 tx=0x7f91b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.489+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b800e040 con 0x7f91cc0fff70 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.489+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91cc111210 con 0x7f91cc0fff70 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.489+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b800bbf0 con 0x7f91cc0fff70 2024-04-16T07:25:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.489+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b801c630 con 0x7f91cc0fff70 2024-04-16T07:25:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.490+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91cc1117d0 con 0x7f91cc0fff70 2024-04-16T07:25:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.491+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91cc0f8ff0 con 0x7f91cc0fff70 2024-04-16T07:25:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.492+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f91b800bd60 con 0x7f91cc0fff70 2024-04-16T07:25:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.494+0000 7f91c3400700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 0x7f91b00711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.494+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f91b808fc20 con 0x7f91cc0fff70 2024-04-16T07:25:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.494+0000 7f91cb400700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 0x7f91b00711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.495+0000 7f91cb400700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 0x7f91b00711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91cc066030 tx=0x7f91bc009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.499+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f91b805c420 con 0x7f91cc0fff70 2024-04-16T07:25:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.791+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91cc111ad0 con 0x7f91b006ed20 2024-04-16T07:25:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.792+0000 7f91c3400700 1 -- 172.21.15.26:0/316427828 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f91cc111ad0 con 0x7f91b006ed20 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:25:25.792 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.795+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 msgr2=0x7f91b00711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.795+0000 7f91d2600700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 0x7f91b00711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91cc066030 tx=0x7f91bc009ee0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.795+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 msgr2=0x7f91cc110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.795+0000 7f91d2600700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009150 tx=0x7f91b8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.795+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 shutdown_connections 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f91b006ed20 0x7f91b00711d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91cc0662f0 0x7f91cc10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 --2- 172.21.15.26:0/316427828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91cc0fff70 0x7f91cc110250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 >> 172.21.15.26:0/316427828 conn(0x7f91cc0f74c0 msgr2=0x7f91cc0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 shutdown_connections 2024-04-16T07:25:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.796+0000 7f91d2600700 1 -- 172.21.15.26:0/316427828 wait complete. 2024-04-16T07:25:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 -- 172.21.15.26:0/1991022229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 msgr2=0x7f762c0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 --2- 172.21.15.26:0/1991022229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c0fe4b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7618004940 tx=0x7f7618009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 -- 172.21.15.26:0/1991022229 shutdown_connections 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 --2- 172.21.15.26:0/1991022229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c0fe4b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 --2- 172.21.15.26:0/1991022229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 0x7f762c0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.958+0000 7f7634000700 1 -- 172.21.15.26:0/1991022229 >> 172.21.15.26:0/1991022229 conn(0x7f762c0f8530 msgr2=0x7f762c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.959+0000 7f7634000700 1 -- 172.21.15.26:0/1991022229 shutdown_connections 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.959+0000 7f7634000700 1 -- 172.21.15.26:0/1991022229 wait complete. 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 Processor -- start 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 -- start start 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 0x7f762c10fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c110780 con 0x7f762c0fe040 2024-04-16T07:25:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.960+0000 7f7634000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c110fd0 con 0x7f762c0fce40 2024-04-16T07:25:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42204/0 (socket says 172.21.15.26:42204) 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 -- 172.21.15.26:0/2855740045 learned_addr learned my addr 172.21.15.26:0/2855740045 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7631600700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 0x7f762c10fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 -- 172.21.15.26:0/2855740045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 msgr2=0x7f762c10fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 0x7f762c10fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 -- 172.21.15.26:0/2855740045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7618005040 con 0x7f762c0fe040 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.961+0000 7f7630c00700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7618009150 tx=0x7f7618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.962+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f761800e040 con 0x7f762c0fe040 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.962+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f761800bbf0 con 0x7f762c0fe040 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.962+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c111250 con 0x7f762c0fe040 2024-04-16T07:25:25.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.962+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f761801c630 con 0x7f762c0fe040 2024-04-16T07:25:25.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.962+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c111790 con 0x7f762c0fe040 2024-04-16T07:25:25.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.963+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c1019d0 con 0x7f762c0fe040 2024-04-16T07:25:25.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.965+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f761800bd60 con 0x7f762c0fe040 2024-04-16T07:25:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.966+0000 7f7629600700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 0x7f76140711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.966+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f761808feb0 con 0x7f762c0fe040 2024-04-16T07:25:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.966+0000 7f7631600700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 0x7f76140711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.967+0000 7f7631600700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 0x7f76140711d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7620009fd0 tx=0x7f7620009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:25.975+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f761805c5f0 con 0x7f762c0fe040 2024-04-16T07:25:26.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:26.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:26.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:26.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:26.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:26.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:26.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:26.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3214206450' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:26.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:26.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:26.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3214206450' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:26.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.334+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.334+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.334+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.335+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:26.335+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:26.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.350+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f762c05f5f0 con 0x7f762c0fe040 2024-04-16T07:25:26.350 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:25:26.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.351+0000 7f7629600700 1 -- 172.21.15.26:0/2855740045 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f761805c180 con 0x7f762c0fe040 2024-04-16T07:25:26.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.353+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 msgr2=0x7f76140711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:26.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.353+0000 7f7634000700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 0x7f76140711d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7620009fd0 tx=0x7f7620009f00 comp rx=0 tx=0).stop 2024-04-16T07:25:26.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.353+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 msgr2=0x7f762c110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:26.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.353+0000 7f7634000700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7618009150 tx=0x7f7618009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:26.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 shutdown_connections 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f761406ed20 0x7f76140711d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f762c0fce40 0x7f762c10fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 --2- 172.21.15.26:0/2855740045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f762c0fe040 0x7f762c110180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 >> 172.21.15.26:0/2855740045 conn(0x7f762c0f8530 msgr2=0x7f762c0f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 shutdown_connections 2024-04-16T07:25:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:26.354+0000 7f7634000700 1 -- 172.21.15.26:0/2855740045 wait complete. 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:27.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:27.046+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:27.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:26 smithi104 ceph-mon[81027]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:27.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:26 smithi104 ceph-mon[81027]: Reconfiguring mon.smithi026 (monmap changed)... 2024-04-16T07:25:27.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:26 smithi104 ceph-mon[81027]: Reconfiguring daemon mon.smithi026 on smithi026 2024-04-16T07:25:27.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:26 smithi104 ceph-mon[81027]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:27.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:26 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2855740045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:25:27.248 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:26 smithi026 ceph-mon[92215]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:27.248 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:26 smithi026 ceph-mon[92215]: Reconfiguring mon.smithi026 (monmap changed)... 2024-04-16T07:25:27.248 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:26 smithi026 ceph-mon[92215]: Reconfiguring daemon mon.smithi026 on smithi026 2024-04-16T07:25:27.248 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:26 smithi026 ceph-mon[92215]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:27.248 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:26 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2855740045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:27.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:27.335+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:28.047+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:28.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:28.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:28.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:28.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:28.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:28.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:28.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:29.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: Reconfiguring mgr.smithi026.tawexg (monmap changed)... 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:29.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:28 smithi026 ceph-mon[92215]: Reconfiguring daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:29.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:29.047+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: Reconfiguring mgr.smithi026.tawexg (monmap changed)... 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.tawexg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:28 smithi104 ceph-mon[81027]: Reconfiguring daemon mgr.smithi026.tawexg on smithi026 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:29.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:29.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:30.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:30.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:30.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:30.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:30.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:30.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:30.047+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:30.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:30.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:30.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:30.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:30.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:30.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:30.335+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:31.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:31.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:31.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:31.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:31.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:31.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:31.047+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:31.222 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:30 smithi026 ceph-mon[92215]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:31.223 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:31.223 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:31.223 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:25:31.223 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:30 smithi104 ceph-mon[81027]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:25:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:31.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:31.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:31.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:31.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:31.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:31.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:31.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.047+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.048+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.048+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.048+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.048+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:32.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:32.048+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:32.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:31 smithi026 ceph-mon[92215]: Reconfiguring crash.smithi026 (monmap changed)... 2024-04-16T07:25:32.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:31 smithi026 ceph-mon[92215]: Reconfiguring daemon crash.smithi026 on smithi026 2024-04-16T07:25:32.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:31 smithi104 ceph-mon[81027]: Reconfiguring crash.smithi026 (monmap changed)... 2024-04-16T07:25:32.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:31 smithi104 ceph-mon[81027]: Reconfiguring daemon crash.smithi026 on smithi026 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 17) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:25:32.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:32.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:32.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 21 ==== client_session(renewcaps seq 17) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:32.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:32.336+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 17) v1 from mds.0 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:33.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:33.048+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:33.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:32 smithi026 ceph-mon[92215]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:33.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:25:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:32 smithi104 ceph-mon[81027]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:25:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:33.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:33.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:34.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:34.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:34.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:34.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:34.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:34.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:34.048+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:34.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:33 smithi026 ceph-mon[92215]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T07:25:34.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:33 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.0 on smithi026 2024-04-16T07:25:34.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:33 smithi104 ceph-mon[81027]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T07:25:34.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:33 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.0 on smithi026 2024-04-16T07:25:34.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:34.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:34.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:34.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:34.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:34.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:34.336+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:35.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:35.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:35.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:35.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:35.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:35.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:35.048+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:35.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:34 smithi026 ceph-mon[92215]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:35.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:34 smithi104 ceph-mon[81027]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:35.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:35.337+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:36.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.048+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:36.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.048+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:36.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.048+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:36.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.048+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:36.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.049+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:36.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:36.049+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:36.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:36.337+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.1 on smithi026 2024-04-16T07:25:36.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:36 smithi026 ceph-mon[92215]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:36.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:36.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:36.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T07:25:36.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:25:36.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:36.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.1 on smithi026 2024-04-16T07:25:36.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:36 smithi104 ceph-mon[81027]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:37.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:37.049+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:37.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:37.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:37.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:37.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:37.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:37.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:37.337+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:38.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:38.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:38.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:38.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:38.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:38.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:38.049+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:38.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:38.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:38.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:38.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:38.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:38.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:38.337+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:38 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.2 on smithi026 2024-04-16T07:25:39.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:39.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:39.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:39.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:39.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:39.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:39.049+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:25:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:39.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:38 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.2 on smithi026 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:39.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:39.338+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:40.049+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:40.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:40.338+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:41.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:40 smithi026 ceph-mon[92215]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:41.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:41.050+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:40 smithi104 ceph-mon[81027]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:41.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:41.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:41.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:41.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:41.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:41.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:41.338+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:42.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:41 smithi026 ceph-mon[92215]: Reconfiguring mds.cephfs.smithi026.kqpyuh (monmap changed)... 2024-04-16T07:25:42.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:41 smithi026 ceph-mon[92215]: Reconfiguring daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:42.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:42.050+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:42.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:41 smithi104 ceph-mon[81027]: Reconfiguring mds.cephfs.smithi026.kqpyuh (monmap changed)... 2024-04-16T07:25:42.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:41 smithi104 ceph-mon[81027]: Reconfiguring daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:25:42.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:42.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:42.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:42.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:42.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:42.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:42.339+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:42 smithi026 ceph-mon[92215]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:43.050+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:42 smithi104 ceph-mon[81027]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:43.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:43.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:43.339+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:44.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:43 smithi026 ceph-mon[92215]: Reconfiguring mds.cephfs.smithi026.kwjewe (monmap changed)... 2024-04-16T07:25:44.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:43 smithi026 ceph-mon[92215]: Reconfiguring daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:25:44.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:44.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 18) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.050+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.051+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 23 ==== client_session(renewcaps seq 18) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:25:44.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:44.051+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 18) v1 from mds.0 2024-04-16T07:25:44.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:43 smithi104 ceph-mon[81027]: Reconfiguring mds.cephfs.smithi026.kwjewe (monmap changed)... 2024-04-16T07:25:44.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:43 smithi104 ceph-mon[81027]: Reconfiguring daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:44.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:44.339+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:45.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:44 smithi026 ceph-mon[92215]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:45.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:45.051+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:45.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:44 smithi104 ceph-mon[81027]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:45.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:45.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:45.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:45.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:45.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:45.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:45.339+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:46.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:46.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:46.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-16T07:25:46.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:25:46.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:46.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:45 smithi104 ceph-mon[81027]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-16T07:25:46.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:46.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:46.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:46.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:46.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:46.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:46.052+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:46.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:45 smithi026 ceph-mon[92215]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:46.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:46.340+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:47.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:47.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:47.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:47.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:47.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:47.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:47.051+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:47.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:46 smithi104 ceph-mon[81027]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:47.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:46 smithi026 ceph-mon[92215]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:47.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:47.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:47.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:47.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:47.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:47.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:47.340+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:48.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:48.052+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: Reconfiguring mgr.smithi104.uqfljg (monmap changed)... 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: Reconfiguring daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:25:48.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:48 smithi104 ceph-mon[81027]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:48.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:48.340+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: Reconfiguring mgr.smithi104.uqfljg (monmap changed)... 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.uqfljg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: Reconfiguring daemon mgr.smithi104.uqfljg on smithi104 2024-04-16T07:25:48.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:48 smithi026 ceph-mon[92215]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:49.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:49.052+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:49.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:49.340+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:50.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:50.052+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:50.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.340+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:50.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:50.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.340+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:50.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.340+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:50.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.340+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:50.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:50.341+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-16T07:25:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:50.525 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:50.525 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:50.525 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-16T07:25:50.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:25:51.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:51.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:51.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:51.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:51.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:51.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:51.052+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:51.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:51.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:51.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:51.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:51.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:51.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:51.341+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:52.053+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 18) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:52.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:25:52.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:52.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.341+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:52.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.342+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 22 ==== client_session(renewcaps seq 18) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:52.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:52.342+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 18) v1 from mds.0 2024-04-16T07:25:52.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:52.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:52.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:52.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T07:25:52.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:25:52.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:52.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:52 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.3 on smithi104 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:52 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.3 on smithi104 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:53.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:53.053+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:53.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:53.341+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:54.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:54.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:54.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:54.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:54.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:54.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:54.054+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:54.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:54.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:54.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:54.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:54.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:54.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:54.341+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:55.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:55.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:55.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:55.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:55.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:55.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:55.053+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:55.068 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:55.068 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:55.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:55.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T07:25:55.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:25:55.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:55.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:54 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.4 on smithi104 2024-04-16T07:25:55.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:54 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.4 on smithi104 2024-04-16T07:25:55.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:55.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:55.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:55.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:55.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:55.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:55.342+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.053+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.053+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.053+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.053+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.054+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:56.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:56.054+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:56.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:56.342+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 -- 172.21.15.26:0/3075006837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fe0e0 msgr2=0x7f36e00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3075006837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fe0e0 0x7f36e00fe550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36c8004940 tx=0x7f36c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 -- 172.21.15.26:0/3075006837 shutdown_connections 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3075006837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fe0e0 0x7f36e00fe550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3075006837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fcef0 0x7f36e00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 -- 172.21.15.26:0/3075006837 >> 172.21.15.26:0/3075006837 conn(0x7f36e00f8530 msgr2=0x7f36e00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:56.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 -- 172.21.15.26:0/3075006837 shutdown_connections 2024-04-16T07:25:56.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.516+0000 7f36e6c00700 1 -- 172.21.15.26:0/3075006837 wait complete. 2024-04-16T07:25:56.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.517+0000 7f36e6c00700 1 Processor -- start 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.517+0000 7f36e6c00700 1 -- start start 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36e6c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36e6c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fe0e0 0x7f36e0110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36e6c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e0110780 con 0x7f36e00fcef0 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36e6c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e0110fd0 con 0x7f36e00fe0e0 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36dfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36dfe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45408/0 (socket says 172.21.15.26:45408) 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.518+0000 7f36dfe00700 1 -- 172.21.15.26:0/3771091154 learned_addr learned my addr 172.21.15.26:0/3771091154 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:56.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36d7e00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fe0e0 0x7f36e0110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36dfe00700 1 -- 172.21.15.26:0/3771091154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fe0e0 msgr2=0x7f36e0110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36dfe00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fe0e0 0x7f36e0110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36dfe00700 1 -- 172.21.15.26:0/3771091154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c8005040 con 0x7f36e00fcef0 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36dfe00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36d0002a80 tx=0x7f36d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36d0007cc0 con 0x7f36e00fcef0 2024-04-16T07:25:56.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.519+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e01112b0 con 0x7f36e00fcef0 2024-04-16T07:25:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.520+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e01117f0 con 0x7f36e00fcef0 2024-04-16T07:25:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.520+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36d000f040 con 0x7f36e00fcef0 2024-04-16T07:25:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.520+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36d000dad0 con 0x7f36e00fcef0 2024-04-16T07:25:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.521+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc0052e0 con 0x7f36e00fcef0 2024-04-16T07:25:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.522+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f36d000dc30 con 0x7f36e00fcef0 2024-04-16T07:25:56.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.523+0000 7f36dcc00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 0x7f36c40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.523+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f36d008e810 con 0x7f36e00fcef0 2024-04-16T07:25:56.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.523+0000 7f36d7e00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 0x7f36c40711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.524+0000 7f36d7e00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 0x7f36c40711d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f36c8000c00 tx=0x7f36c8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:56.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.528+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f36d005aed0 con 0x7f36e00fcef0 2024-04-16T07:25:56.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.782+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36bc000bf0 con 0x7f36c406ed20 2024-04-16T07:25:56.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.784+0000 7f36dcc00700 1 -- 172.21.15.26:0/3771091154 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f36bc000bf0 con 0x7f36c406ed20 2024-04-16T07:25:56.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.788+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 msgr2=0x7f36c40711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.788+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 0x7f36c40711d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f36c8000c00 tx=0x7f36c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.788+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 msgr2=0x7f36e010fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.788+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36d0002a80 tx=0x7f36d0004600 comp rx=0 tx=0).stop 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 shutdown_connections 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36c406ed20 0x7f36c40711d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36e00fcef0 0x7f36e010fc40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 --2- 172.21.15.26:0/3771091154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e00fe0e0 0x7f36e0110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 >> 172.21.15.26:0/3771091154 conn(0x7f36e00f8530 msgr2=0x7f36e00f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 shutdown_connections 2024-04-16T07:25:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.789+0000 7f36e6c00700 1 -- 172.21.15.26:0/3771091154 wait complete. 2024-04-16T07:25:56.803 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:25:56.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 -- 172.21.15.26:0/4198668129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fcef0 msgr2=0x7fb51c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 --2- 172.21.15.26:0/4198668129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fcef0 0x7fb51c0fd320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb50c004940 tx=0x7fb50c009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:56.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 -- 172.21.15.26:0/4198668129 shutdown_connections 2024-04-16T07:25:56.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 --2- 172.21.15.26:0/4198668129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 --2- 172.21.15.26:0/4198668129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fcef0 0x7fb51c0fd320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.944+0000 7fb522000700 1 -- 172.21.15.26:0/4198668129 >> 172.21.15.26:0/4198668129 conn(0x7fb51c0f8530 msgr2=0x7fb51c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:56.944 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:56 smithi026 ceph-mon[92215]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:56.944 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:56.944 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:56.944 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:25:56.945 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.945+0000 7fb522000700 1 -- 172.21.15.26:0/4198668129 shutdown_connections 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.945+0000 7fb522000700 1 -- 172.21.15.26:0/4198668129 wait complete. 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.946+0000 7fb522000700 1 Processor -- start 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.946+0000 7fb522000700 1 -- start start 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.946+0000 7fb522000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fcef0 0x7fb51c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.946+0000 7fb522000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.946+0000 7fb522000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0708c0 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb522000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c070a30 con 0x7fb51c0fcef0 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45426/0 (socket says 172.21.15.26:45426) 2024-04-16T07:25:56.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51aa00700 1 -- 172.21.15.26:0/80341088 learned_addr learned my addr 172.21.15.26:0/80341088 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:56.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51b400700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fcef0 0x7fb51c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51aa00700 1 -- 172.21.15.26:0/80341088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fcef0 msgr2=0x7fb51c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.947+0000 7fb51aa00700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fcef0 0x7fb51c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb51aa00700 1 -- 172.21.15.26:0/80341088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c005040 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb51aa00700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb50800c8c0 tx=0x7fb5080044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb50800a980 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb50800ce60 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5080115e0 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.948+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c070d10 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.949+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c115ab0 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.950+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c05f5f0 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.951+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb508012400 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.952+0000 7fb513400700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 0x7fb5000711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:56.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.952+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb50808eb70 con 0x7fb51c0fe0e0 2024-04-16T07:25:56.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.952+0000 7fb51b400700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 0x7fb5000711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:56.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.953+0000 7fb51b400700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 0x7fb5000711d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb50c004890 tx=0x7fb50c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:56.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:56.957+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb508057810 con 0x7fb51c0fe0e0 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:57.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:57.054+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:56 smithi104 ceph-mon[81027]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:25:57.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:57.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.204+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51c102440 con 0x7fb50006ed20 2024-04-16T07:25:57.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.206+0000 7fb513400700 1 -- 172.21.15.26:0/80341088 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fb51c102440 con 0x7fb50006ed20 2024-04-16T07:25:57.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 msgr2=0x7fb5000711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 0x7fb5000711d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb50c004890 tx=0x7fb50c009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:57.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 msgr2=0x7fb51c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb50800c8c0 tx=0x7fb5080044c0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 shutdown_connections 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb50006ed20 0x7fb5000711d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51c0fcef0 0x7fb51c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 --2- 172.21.15.26:0/80341088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c0fe0e0 0x7fb51c070310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 >> 172.21.15.26:0/80341088 conn(0x7fb51c0f8530 msgr2=0x7fb51c0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 shutdown_connections 2024-04-16T07:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.209+0000 7fb522000700 1 -- 172.21.15.26:0/80341088 wait complete. 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:57.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:57.342+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 -- 172.21.15.26:0/231468109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 msgr2=0x7f609c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 --2- 172.21.15.26:0/231468109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c102390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6088004940 tx=0x7f6088009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 -- 172.21.15.26:0/231468109 shutdown_connections 2024-04-16T07:25:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 --2- 172.21.15.26:0/231468109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c102390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 --2- 172.21.15.26:0/231468109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 -- 172.21.15.26:0/231468109 >> 172.21.15.26:0/231468109 conn(0x7f609c0f74c0 msgr2=0x7f609c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 -- 172.21.15.26:0/231468109 shutdown_connections 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.364+0000 7f60a3800700 1 -- 172.21.15.26:0/231468109 wait complete. 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.365+0000 7f60a3800700 1 Processor -- start 2024-04-16T07:25:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f60a3800700 1 -- start start 2024-04-16T07:25:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f60a3800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f60a3800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f60a3800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c110850 con 0x7f609c0fff70 2024-04-16T07:25:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f60a3800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c110f90 con 0x7f609c0662f0 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f609be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f609be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45446/0 (socket says 172.21.15.26:45446) 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.366+0000 7f609be00700 1 -- 172.21.15.26:0/3318266262 learned_addr learned my addr 172.21.15.26:0/3318266262 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f60a0e00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f609be00700 1 -- 172.21.15.26:0/3318266262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 msgr2=0x7f609c10fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f609be00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c10fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f609be00700 1 -- 172.21.15.26:0/3318266262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088005040 con 0x7f609c0fff70 2024-04-16T07:25:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f60a0e00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.367+0000 7f609be00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f608800b3b0 tx=0x7f6088009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.368+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608800e040 con 0x7f609c0fff70 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.368+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6088004290 con 0x7f609c0fff70 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.368+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c111210 con 0x7f609c0fff70 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.368+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608801c630 con 0x7f609c0fff70 2024-04-16T07:25:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.368+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c1117d0 con 0x7f609c0fff70 2024-04-16T07:25:57.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.369+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c0f8ff0 con 0x7f609c0fff70 2024-04-16T07:25:57.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.370+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f608800bbf0 con 0x7f609c0fff70 2024-04-16T07:25:57.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.371+0000 7f6098c00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 0x7f60840711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.371+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f608808fef0 con 0x7f609c0fff70 2024-04-16T07:25:57.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.372+0000 7f60a0e00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 0x7f60840711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.373+0000 7f60a0e00700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 0x7f60840711d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f609c066030 tx=0x7f609000b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:57.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.376+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f608805c630 con 0x7f609c0fff70 2024-04-16T07:25:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.615+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f609c111b00 con 0x7f608406ed20 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.646+0000 7f6098c00700 1 -- 172.21.15.26:0/3318266262 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f609c111b00 con 0x7f608406ed20 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (2m) 60s ago 13m 14.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (13m) 60s ago 13m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 a7e005cb65de 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (12m) 39s ago 12m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 b3a5d1bb49c1 2024-04-16T07:25:57.645 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (2m) 60s ago 12m 72.4M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (6m) 60s ago 6m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (6m) 60s ago 6m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (6m) 39s ago 6m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (6m) 39s ago 6m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (2m) 60s ago 15m 479M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (98s) 39s ago 11m 449M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (78s) 60s ago 15m 29.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (45s) 39s ago 11m 20.3M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (3m) 60s ago 13m 7894k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (2m) 39s ago 12m 7696k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (10m) 60s ago 10m 46.2M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (10m) 60s ago 10m 49.1M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (9m) 60s ago 9m 46.6M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:25:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (9m) 39s ago 9m 45.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:25:57.647 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (8m) 39s ago 8m 44.5M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:25:57.647 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (8m) 39s ago 8m 43.9M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:25:57.647 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (106s) 60s ago 12m 48.2M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:25:57.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 msgr2=0x7f60840711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 0x7f60840711d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f609c066030 tx=0x7f609000b040 comp rx=0 tx=0).stop 2024-04-16T07:25:57.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 msgr2=0x7f609c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f608800b3b0 tx=0x7f6088009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 shutdown_connections 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f608406ed20 0x7f60840711d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f609c0662f0 0x7f609c10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 --2- 172.21.15.26:0/3318266262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f609c0fff70 0x7f609c110250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 >> 172.21.15.26:0/3318266262 conn(0x7f609c0f74c0 msgr2=0x7f609c0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.649+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 shutdown_connections 2024-04-16T07:25:57.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.650+0000 7f60a3800700 1 -- 172.21.15.26:0/3318266262 wait complete. 2024-04-16T07:25:57.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 -- 172.21.15.26:0/3610117475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f40ffda0 msgr2=0x7f31f41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 --2- 172.21.15.26:0/3610117475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f40ffda0 0x7f31f41021c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31e0004940 tx=0x7f31e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 -- 172.21.15.26:0/3610117475 shutdown_connections 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 --2- 172.21.15.26:0/3610117475 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f40ffda0 0x7f31f41021c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 --2- 172.21.15.26:0/3610117475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f4066cc0 0x7f31f40ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 -- 172.21.15.26:0/3610117475 >> 172.21.15.26:0/3610117475 conn(0x7f31f40f72d0 msgr2=0x7f31f40f9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 -- 172.21.15.26:0/3610117475 shutdown_connections 2024-04-16T07:25:57.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.804+0000 7f31f3e00700 1 -- 172.21.15.26:0/3610117475 wait complete. 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.805+0000 7f31f3e00700 1 Processor -- start 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f3e00700 1 -- start start 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f3e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f40ffda0 0x7f31f410ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f3e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f41105c0 con 0x7f31f4066cc0 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f3e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f4110e10 con 0x7f31f40ffda0 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f2a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45468/0 (socket says 172.21.15.26:45468) 2024-04-16T07:25:57.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.806+0000 7f31f2a00700 1 -- 172.21.15.26:0/1272338641 learned_addr learned my addr 172.21.15.26:0/1272338641 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f2000700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f40ffda0 0x7f31f410ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f2a00700 1 -- 172.21.15.26:0/1272338641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f40ffda0 msgr2=0x7f31f410ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f2a00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f40ffda0 0x7f31f410ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f2a00700 1 -- 172.21.15.26:0/1272338641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0005040 con 0x7f31f4066cc0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f2a00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31e4002a50 tx=0x7f31e4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e4007cc0 con 0x7f31f4066cc0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31e400f040 con 0x7f31f4066cc0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e400daf0 con 0x7f31f4066cc0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f41110f0 con 0x7f31f4066cc0 2024-04-16T07:25:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.807+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f41115e0 con 0x7f31f4066cc0 2024-04-16T07:25:57.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.809+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f40faf70 con 0x7f31f4066cc0 2024-04-16T07:25:57.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.810+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f31e4009810 con 0x7f31f4066cc0 2024-04-16T07:25:57.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.811+0000 7f31eaa00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 0x7f31d80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:57.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.811+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31e408e8a0 con 0x7f31f4066cc0 2024-04-16T07:25:57.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.811+0000 7f31f2000700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 0x7f31d80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:57.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.812+0000 7f31f2000700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 0x7f31d80711e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31e0004f90 tx=0x7f31e0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:57.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:57.815+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f31e405af60 con 0x7f31f4066cc0 2024-04-16T07:25:58.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:57 smithi026 ceph-mon[92215]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T07:25:58.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:57 smithi026 ceph-mon[92215]: Reconfiguring daemon osd.5 on smithi104 2024-04-16T07:25:58.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:57 smithi026 ceph-mon[92215]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:58.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:58.054+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:57 smithi104 ceph-mon[81027]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T07:25:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:57 smithi104 ceph-mon[81027]: Reconfiguring daemon osd.5 on smithi104 2024-04-16T07:25:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:57 smithi104 ceph-mon[81027]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:58.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.132+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f31f4105770 con 0x7f31f4066cc0 2024-04-16T07:25:58.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.133+0000 7f31eaa00700 1 -- 172.21.15.26:0/1272338641 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f31e405aaf0 con 0x7f31f4066cc0 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:25:58.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:25:58.134 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:25:58.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 msgr2=0x7f31d80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 0x7f31d80711e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31e0004f90 tx=0x7f31e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:58.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 msgr2=0x7f31f410fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31e4002a50 tx=0x7f31e4004600 comp rx=0 tx=0).stop 2024-04-16T07:25:58.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 shutdown_connections 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f31d806ed30 0x7f31d80711e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31f4066cc0 0x7f31f410fa80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 --2- 172.21.15.26:0/1272338641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31f40ffda0 0x7f31f410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 >> 172.21.15.26:0/1272338641 conn(0x7f31f40f72d0 msgr2=0x7f31f40fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 shutdown_connections 2024-04-16T07:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.137+0000 7f31f3e00700 1 -- 172.21.15.26:0/1272338641 wait complete. 2024-04-16T07:25:58.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 -- 172.21.15.26:0/1960765086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 msgr2=0x7f0c3c0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 --2- 172.21.15.26:0/1960765086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c0fd290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c004940 tx=0x7f0c2c009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 -- 172.21.15.26:0/1960765086 shutdown_connections 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 --2- 172.21.15.26:0/1960765086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 0x7f0c3c0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 --2- 172.21.15.26:0/1960765086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c0fd290 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 -- 172.21.15.26:0/1960765086 >> 172.21.15.26:0/1960765086 conn(0x7f0c3c0f8580 msgr2=0x7f0c3c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 -- 172.21.15.26:0/1960765086 shutdown_connections 2024-04-16T07:25:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.288+0000 7f0c41c00700 1 -- 172.21.15.26:0/1960765086 wait complete. 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.289+0000 7f0c41c00700 1 Processor -- start 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.289+0000 7f0c41c00700 1 -- start start 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c41c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c41c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 0x7f0c3c10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c41c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c10e470 con 0x7f0c3c0fce80 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c41c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c10e5e0 con 0x7f0c3c0fdff0 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60632/0 (socket says 172.21.15.26:60632) 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 -- 172.21.15.26:0/2768076509 learned_addr learned my addr 172.21.15.26:0/2768076509 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:58.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3a000700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 0x7f0c3c10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 -- 172.21.15.26:0/2768076509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 msgr2=0x7f0c3c10de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 0x7f0c3c10de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.290+0000 7f0c3aa00700 1 -- 172.21.15.26:0/2768076509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c005040 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c3aa00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00b3b0 tx=0x7f0c2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2c00e040 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c2c004290 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2c01c630 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c112fe0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.291+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c1134c0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.292+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c04c710 con 0x7f0c3c0fce80 2024-04-16T07:25:58.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.293+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f0c2c00bbf0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.294+0000 7f0c32a00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 0x7f0c200711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.294+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0c2c08ffc0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.294+0000 7f0c3a000700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 0x7f0c200711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.295+0000 7f0c3a000700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 0x7f0c200711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c28007fd0 tx=0x7f0c28009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:58.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.299+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0c2c0c4ff0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:58.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:58.342+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:58.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.578+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0c3c1139b0 con 0x7f0c3c0fce80 2024-04-16T07:25:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.579+0000 7f0c32a00700 1 -- 172.21.15.26:0/2768076509 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f0c2c05c680 con 0x7f0c3c0fce80 2024-04-16T07:25:58.578 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:25:58.579 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:25:58.580 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:25:58.581 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:58.582 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:58.582 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 msgr2=0x7f0c200711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 0x7f0c200711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c28007fd0 tx=0x7f0c28009040 comp rx=0 tx=0).stop 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 msgr2=0x7f0c3c10d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00b3b0 tx=0x7f0c2c009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 shutdown_connections 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0c2006ed30 0x7f0c200711e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c3c0fce80 0x7f0c3c10d930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.581+0000 7f0c41c00700 1 --2- 172.21.15.26:0/2768076509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c3c0fdff0 0x7f0c3c10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.582+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 >> 172.21.15.26:0/2768076509 conn(0x7f0c3c0f8580 msgr2=0x7f0c3c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.582+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 shutdown_connections 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.582+0000 7f0c41c00700 1 -- 172.21.15.26:0/2768076509 wait complete. 2024-04-16T07:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:25:58.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 -- 172.21.15.26:0/2161248462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574070890 msgr2=0x7f0574070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 --2- 172.21.15.26:0/2161248462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574070890 0x7f0574070d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0560004940 tx=0x7f0560009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 -- 172.21.15.26:0/2161248462 shutdown_connections 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 --2- 172.21.15.26:0/2161248462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574070890 0x7f0574070d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 --2- 172.21.15.26:0/2161248462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574071e40 0x7f0574070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 -- 172.21.15.26:0/2161248462 >> 172.21.15.26:0/2161248462 conn(0x7f05740f8580 msgr2=0x7f05740fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 -- 172.21.15.26:0/2161248462 shutdown_connections 2024-04-16T07:25:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.737+0000 7f057c000700 1 -- 172.21.15.26:0/2161248462 wait complete. 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.738+0000 7f057c000700 1 Processor -- start 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.738+0000 7f057c000700 1 -- start start 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f057c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574070890 0x7f057410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f057c000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f057c000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0574110730 con 0x7f0574071e40 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f057c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05741150d0 con 0x7f0574070890 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f0578c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f0578c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60662/0 (socket says 172.21.15.26:60662) 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f0578c00700 1 -- 172.21.15.26:0/286403924 learned_addr learned my addr 172.21.15.26:0/286403924 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.739+0000 7f0579600700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574070890 0x7f057410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0578c00700 1 -- 172.21.15.26:0/286403924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574070890 msgr2=0x7f057410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0578c00700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574070890 0x7f057410fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0578c00700 1 -- 172.21.15.26:0/286403924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560005040 con 0x7f0574071e40 2024-04-16T07:25:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0578c00700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0560009150 tx=0x7f0560009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056000e040 con 0x7f0574071e40 2024-04-16T07:25:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f056000bcc0 con 0x7f0574071e40 2024-04-16T07:25:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05741152d0 con 0x7f0574071e40 2024-04-16T07:25:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056001c630 con 0x7f0574071e40 2024-04-16T07:25:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.740+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0574115840 con 0x7f0574071e40 2024-04-16T07:25:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.741+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057404c710 con 0x7f0574071e40 2024-04-16T07:25:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.743+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f0560013420 con 0x7f0574071e40 2024-04-16T07:25:58.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.743+0000 7f0571600700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 0x7f055c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:58.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.744+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f056008f7b0 con 0x7f0574071e40 2024-04-16T07:25:58.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.744+0000 7f0579600700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 0x7f055c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:58.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.745+0000 7f0579600700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 0x7f055c0711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0568006fd0 tx=0x7f0568008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:58.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.748+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f056005bef0 con 0x7f0574071e40 2024-04-16T07:25:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:58.999+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0574103e60 con 0x7f055c06ed30 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1272338641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:58.999 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:59.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:59.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:59.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:58 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2768076509' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:25:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.003+0000 7f0571600700 1 -- 172.21.15.26:0/286403924 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f0574103e60 con 0x7f055c06ed30 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T07:25:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T07:25:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:25:59.003 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:25:59.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.005+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 msgr2=0x7f055c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.005+0000 7f057c000700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 0x7f055c0711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0568006fd0 tx=0x7f0568008040 comp rx=0 tx=0).stop 2024-04-16T07:25:59.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.005+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 msgr2=0x7f05741100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.005+0000 7f057c000700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0560009150 tx=0x7f0560009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 shutdown_connections 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f055c06ed30 0x7f055c0711e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0574070890 0x7f057410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 --2- 172.21.15.26:0/286403924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0574071e40 0x7f05741100a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 >> 172.21.15.26:0/286403924 conn(0x7f05740f8580 msgr2=0x7f05740f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 shutdown_connections 2024-04-16T07:25:59.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.006+0000 7f057c000700 1 -- 172.21.15.26:0/286403924 wait complete. 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:25:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:25:59.054+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1272338641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:25:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:58 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2768076509' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 -- 172.21.15.26:0/3350670216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c103420 msgr2=0x7f138c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 --2- 172.21.15.26:0/3350670216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c103420 0x7f138c103830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1380004940 tx=0x7f1380009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 -- 172.21.15.26:0/3350670216 shutdown_connections 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 --2- 172.21.15.26:0/3350670216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c1046d0 0x7f138c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 --2- 172.21.15.26:0/3350670216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c103420 0x7f138c103830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 -- 172.21.15.26:0/3350670216 >> 172.21.15.26:0/3350670216 conn(0x7f138c072640 msgr2=0x7f138c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:59.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 -- 172.21.15.26:0/3350670216 shutdown_connections 2024-04-16T07:25:59.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.162+0000 7f1394800700 1 -- 172.21.15.26:0/3350670216 wait complete. 2024-04-16T07:25:59.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 Processor -- start 2024-04-16T07:25:59.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 -- start start 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c1046d0 0x7f138c114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c114a10 con 0x7f138c103420 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1394800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c114b80 con 0x7f138c1046d0 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1391e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1391e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60680/0 (socket says 172.21.15.26:60680) 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.163+0000 7f1391e00700 1 -- 172.21.15.26:0/4001117687 learned_addr learned my addr 172.21.15.26:0/4001117687 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1391400700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c1046d0 0x7f138c114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1391e00700 1 -- 172.21.15.26:0/4001117687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c1046d0 msgr2=0x7f138c114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1391e00700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c1046d0 0x7f138c114410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1391e00700 1 -- 172.21.15.26:0/4001117687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1380005040 con 0x7f138c103420 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1391e00700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f138000b0d0 tx=0x7f1380009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138000e070 con 0x7f138c103420 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f138000b410 con 0x7f138c103420 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.164+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13800125e0 con 0x7f138c103420 2024-04-16T07:25:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.165+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c119580 con 0x7f138c103420 2024-04-16T07:25:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.165+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c119af0 con 0x7f138c103420 2024-04-16T07:25:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.166+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c04c710 con 0x7f138c103420 2024-04-16T07:25:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.167+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f138000b580 con 0x7f138c103420 2024-04-16T07:25:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.168+0000 7f1386000700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 0x7f13740711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:25:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.168+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f138008f6f0 con 0x7f138c103420 2024-04-16T07:25:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.168+0000 7f1391400700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 0x7f13740711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:25:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.169+0000 7f1391400700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 0x7f13740711e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f137c009b80 tx=0x7f137c00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:25:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.173+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f138005bef0 con 0x7f138c103420 2024-04-16T07:25:59.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.342+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:25:59.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:25:59.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.342+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:25:59.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.342+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:25:59.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.343+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:25:59.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:25:59.343+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:25:59.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.491+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f138c10bf10 con 0x7f138c103420 2024-04-16T07:25:59.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.491+0000 7f1386000700 1 -- 172.21.15.26:0/4001117687 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f138c10bf10 con 0x7f138c103420 2024-04-16T07:25:59.491 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 msgr2=0x7f13740711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 0x7f13740711e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f137c009b80 tx=0x7f137c00b040 comp rx=0 tx=0).stop 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 msgr2=0x7f138c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f138000b0d0 tx=0x7f1380009e30 comp rx=0 tx=0).stop 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 shutdown_connections 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f137406ed30 0x7f13740711e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f138c103420 0x7f138c113ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 --2- 172.21.15.26:0/4001117687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f138c1046d0 0x7f138c114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.494+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 >> 172.21.15.26:0/4001117687 conn(0x7f138c072640 msgr2=0x7f138c073c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.495+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 shutdown_connections 2024-04-16T07:25:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:25:59.495+0000 7f1394800700 1 -- 172.21.15.26:0/4001117687 wait complete. 2024-04-16T07:25:59.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:59 smithi104 ceph-mon[81027]: Reconfiguring mds.cephfs.smithi104.dpfdje (monmap changed)... 2024-04-16T07:25:59.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:59 smithi104 ceph-mon[81027]: Reconfiguring daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:25:59.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:59 smithi104 ceph-mon[81027]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:25:59.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:25:59 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4001117687' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:26:00.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:00.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:00.055+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:00.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:59 smithi026 ceph-mon[92215]: Reconfiguring mds.cephfs.smithi104.dpfdje (monmap changed)... 2024-04-16T07:26:00.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:59 smithi026 ceph-mon[92215]: Reconfiguring daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:26:00.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:59 smithi026 ceph-mon[92215]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:00.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:25:59 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4001117687' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:00.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:00.343+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:01.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:01.055+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:00 smithi104 ceph-mon[81027]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:26:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:01.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:00 smithi026 ceph-mon[92215]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:26:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:01.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:01.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:01.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:01.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:01.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:01.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:01.343+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:02.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:02.055+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:02.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:01 smithi104 ceph-mon[81027]: Reconfiguring mds.cephfs.smithi104.zwhiwr (monmap changed)... 2024-04-16T07:26:02.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:01 smithi104 ceph-mon[81027]: Reconfiguring daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:26:02.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:01 smithi026 ceph-mon[92215]: Reconfiguring mds.cephfs.smithi104.zwhiwr (monmap changed)... 2024-04-16T07:26:02.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:01 smithi026 ceph-mon[92215]: Reconfiguring daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:02.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:02.343+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:03.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:03.055+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:02 smithi104 ceph-mon[81027]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:03.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:02 smithi026 ceph-mon[92215]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:03.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:03.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:03.344+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:04.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 19) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:04.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.055+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:04.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.056+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 24 ==== client_session(renewcaps seq 19) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:26:04.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:04.056+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 19) v1 from mds.0 2024-04-16T07:26:04.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:04.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all mon 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi026"}]: dispatch 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi026"}]': finished 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]: dispatch 2024-04-16T07:26:04.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]': finished 2024-04-16T07:26:04.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all mon 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi026"}]: dispatch 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi026"}]': finished 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]: dispatch 2024-04-16T07:26:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]': finished 2024-04-16T07:26:04.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:04.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:04.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:04.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:04.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:04.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:04.344+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:05.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:05.056+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:05.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:04 smithi026 ceph-mon[92215]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:04 smithi104 ceph-mon[81027]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:05.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:05.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:05.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:05.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:05.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:05.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:05.344+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:06.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:06.056+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:06.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:06.344+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:06.675 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:06.676 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: Upgrade: Updating crash.smithi026 (1/2) 2024-04-16T07:26:06.676 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:06.676 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:26:06.676 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:06.676 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:06 smithi026 ceph-mon[92215]: Deploying daemon crash.smithi026 on smithi026 2024-04-16T07:26:06.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:06.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: Upgrade: Updating crash.smithi026 (1/2) 2024-04-16T07:26:06.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:06.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:26:06.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:06.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:06 smithi104 ceph-mon[81027]: Deploying daemon crash.smithi026 on smithi026 2024-04-16T07:26:07.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:07.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:07.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:07.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:07.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:07.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:07.056+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:07.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:07.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:07.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:07.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:07.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:07.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:07.345+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:08.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:08.056+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:08.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:08.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:08.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:08.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:08.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:08.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:08.345+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:08.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:08 smithi026 ceph-mon[92215]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:08.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:08 smithi104 ceph-mon[81027]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:09.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:09.057+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:09.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:09.345+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:10.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:10.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:10.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:10.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:10.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:10.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:10.057+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:10.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:10.345+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:10.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:10 smithi026 ceph-mon[92215]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:10 smithi104 ceph-mon[81027]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:11.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:11.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:11.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:11.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:11.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:11.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:11.057+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:11.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:11.345+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:12.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:12.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:12.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:12.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:12.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:12.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:12.057+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:12.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:12 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:12.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:12 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:12.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:12 smithi104 ceph-mon[81027]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:12.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:12.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 19) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.346+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.347+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 23 ==== client_session(renewcaps seq 19) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:12.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:12.347+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 19) v1 from mds.0 2024-04-16T07:26:12.428 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:12 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:12.429 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:12 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:12.429 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:12 smithi026 ceph-mon[92215]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.057+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.057+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.057+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.058+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.058+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:13.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:13.058+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:13.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:13.346+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:14.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:14.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:14.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:14.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:14.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:14.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:14.058+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:14.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:14.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:14.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:14.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:14.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:14.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:14.346+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:14.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: Upgrade: Updating crash.smithi104 (2/2) 2024-04-16T07:26:14.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:14.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:26:14.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:14.415 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: Deploying daemon crash.smithi104 on smithi104 2024-04-16T07:26:14.415 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:14 smithi104 ceph-mon[81027]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: Upgrade: Updating crash.smithi104 (2/2) 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: Deploying daemon crash.smithi104 on smithi104 2024-04-16T07:26:14.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:14 smithi026 ceph-mon[92215]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:15.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:15.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:15.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:15.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:15.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:15.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:15.058+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:15.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:15.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:15.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:15.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:15.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:15.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:15.346+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:16.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:16.058+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:16.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:16 smithi104 ceph-mon[81027]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:16.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:16.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:16.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:16.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:16.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:16.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:16.347+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:16.643 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:16 smithi026 ceph-mon[92215]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:17.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:17.059+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:17.347+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:18.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:18.059+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:18.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:18.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:18.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:18.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:18.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:18.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:18.347+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:18 smithi104 ceph-mon[81027]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:18.659 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:18 smithi026 ceph-mon[92215]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:19.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:19.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:19.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:19.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:19.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:19.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:19.059+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:19.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:19.347+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:19.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:19.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:19.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:19.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:20.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:20.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:20.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:20.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:20.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:20.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:20.059+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:20.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:20.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:20.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:20.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:20.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:20.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:20.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:20 smithi026 ceph-mon[92215]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:20.763 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:20 smithi104 ceph-mon[81027]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:21.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:21.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:21.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:21.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:21.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:21.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:21.060+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:21.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:21.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:22.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:22.060+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:22.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:22.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:22.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:22.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:22.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:22.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:22.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:22 smithi026 ceph-mon[92215]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:22 smithi026 ceph-mon[92215]: mgrmap e37: smithi026.tawexg(active, since 92s), standbys: smithi104.uqfljg 2024-04-16T07:26:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:22 smithi104 ceph-mon[81027]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:22 smithi104 ceph-mon[81027]: mgrmap e37: smithi026.tawexg(active, since 92s), standbys: smithi104.uqfljg 2024-04-16T07:26:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:23.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.024+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c0040a0 con 0x7fbf70008390 2024-04-16T07:26:23.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.025+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c0040a0 con 0x7fbf70008390 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:23.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:23.060+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:23.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:23.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:24.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 20) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.060+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:24.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.061+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 25 ==== client_session(renewcaps seq 20) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:26:24.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:24.061+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 20) v1 from mds.0 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:24.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:24.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:24 smithi104 ceph-mon[81027]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:24.983 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:24 smithi026 ceph-mon[92215]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.060+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.060+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.060+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.060+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.061+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:25.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:25.061+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:25.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:25.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:25.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:25.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:25.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:25.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:25.348+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:25.672 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:25.672 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:25 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:25.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:25.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:25 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:26.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:26.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:26.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:26.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:26.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:26.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:26.061+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:26.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:26.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:26.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:26.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:26.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:26.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:26.349+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:27.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:27.061+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:27.209 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:26 smithi104 ceph-mon[81027]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:27.209 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:26 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:27.209 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:26 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:27.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:26 smithi026 ceph-mon[92215]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:27.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:26 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:27.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:26 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:27.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:27.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:27.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:27.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 143 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:27.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:27.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:27.349+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:28.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:28.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:28.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:28.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:28.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:28.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:28.061+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86589 2024-04-16T07:26:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 292, success, outsize: 120 2024-04-16T07:26:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.339+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.339+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.339+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.339+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.339+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:28.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:28.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:28.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 144 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:28.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:28.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.349+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:28.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86677 2024-04-16T07:26:28.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 294, success, outsize: 120 2024-04-16T07:26:28.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.359+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.359+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.359+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.359+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.359+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86768 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 296, success, outsize: 120 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.377+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.377+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.377+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.377+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.377+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86865 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 298, success, outsize: 120 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.398+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.398+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.398+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.398+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.398+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86959 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 300, success, outsize: 120 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.422+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.422+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.422+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.422+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.422+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87047 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 302, success, outsize: 120 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.446+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.446+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.446+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.446+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.446+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87138 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 304, success, outsize: 120 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.470+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.470+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.470+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.470+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.470+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87235 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 306, success, outsize: 120 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.496+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.496+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.496+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.496+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.496+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87329 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 308, success, outsize: 120 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.523+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.523+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.523+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.523+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.523+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87417 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 310, success, outsize: 120 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.548+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.548+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.548+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.548+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.548+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87508 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 312, success, outsize: 120 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.574+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.574+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.574+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.574+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.574+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87605 2024-04-16T07:26:28.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 314, success, outsize: 120 2024-04-16T07:26:28.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.599+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.599+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.599+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.599+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.599+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87699 2024-04-16T07:26:28.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 316, success, outsize: 120 2024-04-16T07:26:28.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.626+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.626+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.626+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.626+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.626+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87787 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.651+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 318, success, outsize: 120 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.651+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.651+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.651+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.651+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87878 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 320, success, outsize: 120 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.678+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.678+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.678+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.678+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.678+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87975 2024-04-16T07:26:28.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 322, success, outsize: 120 2024-04-16T07:26:28.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.706+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.706+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.706+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.706+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.706+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88069 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 324, success, outsize: 120 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.734+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.734+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.734+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.734+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.734+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88157 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.760+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 326, success, outsize: 120 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.760+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.760+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.760+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.760+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88248 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 328, success, outsize: 120 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.788+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.788+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.788+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.788+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.788+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88345 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 330, success, outsize: 120 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.818+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.818+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.818+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.818+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.818+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88439 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 332, success, outsize: 120 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.839+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.839+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.839+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.839+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.839+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88527 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 334, success, outsize: 120 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.862+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.862+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.862+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.862+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.862+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88618 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 336, success, outsize: 120 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.888+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.888+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.888+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.888+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.888+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88715 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 338, success, outsize: 120 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.915+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.915+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.915+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.915+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.915+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88809 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 340, success, outsize: 120 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.942+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.942+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.942+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.942+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.942+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88897 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 342, success, outsize: 120 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.967+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.967+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.967+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.967+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.967+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:28.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88988 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.991+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 344, success, outsize: 120 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.991+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.991+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.991+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:28.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:28.991+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89085 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.016+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 346, success, outsize: 120 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.016+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.016+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.016+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:26:29.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.016+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:26:29.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:29.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:29.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:29.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:29.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:29.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:29.061+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:28 smithi026 ceph-mon[92215]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:28 smithi104 ceph-mon[81027]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:29.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:29.349+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.664+0000 7fc6c5400700 1 -- 172.21.15.26:0/3509701111 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 msgr2=0x7fc6c00ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.664+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3509701111 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c00ff430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0004940 tx=0x7fc6b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 -- 172.21.15.26:0/3509701111 shutdown_connections 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3509701111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 0x7fc6c0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3509701111 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c00ff430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 -- 172.21.15.26:0/3509701111 >> 172.21.15.26:0/3509701111 conn(0x7fc6c00fa950 msgr2=0x7fc6c00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 -- 172.21.15.26:0/3509701111 shutdown_connections 2024-04-16T07:26:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 -- 172.21.15.26:0/3509701111 wait complete. 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.665+0000 7fc6c5400700 1 Processor -- start 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6c5400700 1 -- start start 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6c5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6c5400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 0x7fc6c01100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6c5400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c01106a0 con 0x7fc6c00ff020 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6c5400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0110810 con 0x7fc6c01002d0 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6be800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6be800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49318/0 (socket says 172.21.15.26:49318) 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.666+0000 7fc6be800700 1 -- 172.21.15.26:0/3116731593 learned_addr learned my addr 172.21.15.26:0/3116731593 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6bde00700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 0x7fc6c01100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6be800700 1 -- 172.21.15.26:0/3116731593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 msgr2=0x7fc6c01100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6be800700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 0x7fc6c01100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6be800700 1 -- 172.21.15.26:0/3116731593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b0005040 con 0x7fc6c00ff020 2024-04-16T07:26:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6be800700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000b3b0 tx=0x7fc6b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b000e040 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6b0004290 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b001c630 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.667+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c0115210 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.668+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c01157d0 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.669+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c004c710 con 0x7fc6c00ff020 2024-04-16T07:26:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.670+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc6b000bbf0 con 0x7fc6c00ff020 2024-04-16T07:26:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.671+0000 7fc6b6a00700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 0x7fc6a40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.671+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc6b008fbb0 con 0x7fc6c00ff020 2024-04-16T07:26:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.671+0000 7fc6bde00700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 0x7fc6a40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.672+0000 7fc6bde00700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 0x7fc6a40710b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac004c40 tx=0x7fc6ac009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.676+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc6b0096080 con 0x7fc6c00ff020 2024-04-16T07:26:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.931+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6c0104690 con 0x7fc6a406ec00 2024-04-16T07:26:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.933+0000 7fc6b6a00700 1 -- 172.21.15.26:0/3116731593 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7fc6c0104690 con 0x7fc6a406ec00 2024-04-16T07:26:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 msgr2=0x7fc6a40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 0x7fc6a40710b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac004c40 tx=0x7fc6ac009040 comp rx=0 tx=0).stop 2024-04-16T07:26:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 msgr2=0x7fc6c010fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000b3b0 tx=0x7fc6b0009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 shutdown_connections 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc6a406ec00 0x7fc6a40710b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6c00ff020 0x7fc6c010fb60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 --2- 172.21.15.26:0/3116731593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6c01002d0 0x7fc6c01100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.936+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 >> 172.21.15.26:0/3116731593 conn(0x7fc6c00fa950 msgr2=0x7fc6c00fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.937+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 shutdown_connections 2024-04-16T07:26:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:29.937+0000 7fc6c5400700 1 -- 172.21.15.26:0/3116731593 wait complete. 2024-04-16T07:26:29.951 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:30.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:30.062+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 -- 172.21.15.26:0/3229970422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fcff0 msgr2=0x7f3c6c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3229970422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c0fd460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3c54004940 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 -- 172.21.15.26:0/3229970422 shutdown_connections 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3229970422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c0fd460 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3229970422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c0fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 -- 172.21.15.26:0/3229970422 >> 172.21.15.26:0/3229970422 conn(0x7f3c6c0f74c0 msgr2=0x7f3c6c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 -- 172.21.15.26:0/3229970422 shutdown_connections 2024-04-16T07:26:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.091+0000 7f3c72c00700 1 -- 172.21.15.26:0/3229970422 wait complete. 2024-04-16T07:26:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.092+0000 7f3c72c00700 1 Processor -- start 2024-04-16T07:26:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.092+0000 7f3c72c00700 1 -- start start 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c72c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c72c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c72c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c102900 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c72c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c102a70 con 0x7f3c6c0fcff0 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c6be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c6be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49348/0 (socket says 172.21.15.26:49348) 2024-04-16T07:26:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.093+0000 7f3c6be00700 1 -- 172.21.15.26:0/3902542427 learned_addr learned my addr 172.21.15.26:0/3902542427 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c63400700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c6be00700 1 -- 172.21.15.26:0/3902542427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fcff0 msgr2=0x7f3c6c102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c6be00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c102350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c6be00700 1 -- 172.21.15.26:0/3902542427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54005040 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c6be00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c002a50 tx=0x7f3c5c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c007cc0 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c5c00f040 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c009640 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c102d50 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.094+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c115a50 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.096+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c05f5f0 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.098+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3c5c0097c0 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.099+0000 7f3c68c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 0x7f3c500710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.099+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c5c08eb50 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.100+0000 7f3c63400700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 0x7f3c500710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.100+0000 7f3c63400700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 0x7f3c500710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400b970 tx=0x7f3c54009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.103+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3c5c08ee60 con 0x7f3c6c0fbd40 2024-04-16T07:26:30.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:30.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:30.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:30.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:30.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:30.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:30.349+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:30.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.360+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c6c101350 con 0x7f3c5006ec10 2024-04-16T07:26:30.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.362+0000 7f3c68c00700 1 -- 172.21.15.26:0/3902542427 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7f3c6c101350 con 0x7f3c5006ec10 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.364+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 msgr2=0x7f3c500710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.364+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 0x7f3c500710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400b970 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.364+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 msgr2=0x7f3c6c103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.364+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c002a50 tx=0x7f3c5c0046b0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 shutdown_connections 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f3c5006ec10 0x7f3c500710c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c6c0fbd40 0x7f3c6c103e80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 --2- 172.21.15.26:0/3902542427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c6c0fcff0 0x7f3c6c102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 >> 172.21.15.26:0/3902542427 conn(0x7f3c6c0f74c0 msgr2=0x7f3c6c0f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 shutdown_connections 2024-04-16T07:26:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.365+0000 7f3c72c00700 1 -- 172.21.15.26:0/3902542427 wait complete. 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 -- 172.21.15.26:0/3890237281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fe0e0 msgr2=0x7fcf640fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 --2- 172.21.15.26:0/3890237281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fe0e0 0x7fcf640fe550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcf50004940 tx=0x7fcf50009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 -- 172.21.15.26:0/3890237281 shutdown_connections 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 --2- 172.21.15.26:0/3890237281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fe0e0 0x7fcf640fe550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 --2- 172.21.15.26:0/3890237281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fcef0 0x7fcf640fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.524+0000 7fcf6b000700 1 -- 172.21.15.26:0/3890237281 >> 172.21.15.26:0/3890237281 conn(0x7fcf640f8530 msgr2=0x7fcf640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.525+0000 7fcf6b000700 1 -- 172.21.15.26:0/3890237281 shutdown_connections 2024-04-16T07:26:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.525+0000 7fcf6b000700 1 -- 172.21.15.26:0/3890237281 wait complete. 2024-04-16T07:26:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 Processor -- start 2024-04-16T07:26:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 -- start start 2024-04-16T07:26:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fe0e0 0x7fcf64070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf640708a0 con 0x7fcf640fcef0 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.526+0000 7fcf6b000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf640709e0 con 0x7fcf640fe0e0 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49378/0 (socket says 172.21.15.26:49378) 2024-04-16T07:26:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 -- 172.21.15.26:0/1746927421 learned_addr learned my addr 172.21.15.26:0/1746927421 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63400700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fe0e0 0x7fcf64070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 -- 172.21.15.26:0/1746927421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fe0e0 msgr2=0x7fcf64070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fe0e0 0x7fcf64070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 -- 172.21.15.26:0/1746927421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50005040 con 0x7fcf640fcef0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.527+0000 7fcf63e00700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf54002a80 tx=0x7fcf54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.528+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf54007cc0 con 0x7fcf640fcef0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.528+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf5400f040 con 0x7fcf640fcef0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.528+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf5400dad0 con 0x7fcf640fcef0 2024-04-16T07:26:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.528+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf64070cc0 con 0x7fcf640fcef0 2024-04-16T07:26:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.528+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf64115ac0 con 0x7fcf640fcef0 2024-04-16T07:26:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.529+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6405f5f0 con 0x7fcf640fcef0 2024-04-16T07:26:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.530+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf5400dc30 con 0x7fcf640fcef0 2024-04-16T07:26:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.531+0000 7fcf5be00700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 0x7fcf480710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.531+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcf5408e580 con 0x7fcf640fcef0 2024-04-16T07:26:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.531+0000 7fcf63400700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 0x7fcf480710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.532+0000 7fcf63400700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 0x7fcf480710b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000b970 tx=0x7fcf50009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.540+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcf540573b0 con 0x7fcf640fcef0 2024-04-16T07:26:30.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.789+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf641024a0 con 0x7fcf4806ec00 2024-04-16T07:26:30.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.806+0000 7fcf5be00700 1 -- 172.21.15.26:0/1746927421 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fcf641024a0 con 0x7fcf4806ec00 2024-04-16T07:26:30.805 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:26:30.805 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (3m) 4s ago 13m 14.2M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (20s) 4s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (12s) 8s ago 12m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (3m) 4s ago 13m 71.8M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (6m) 4s ago 6m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (6m) 4s ago 6m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (6m) 8s ago 6m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (6m) 8s ago 6m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (3m) 4s ago 15m 497M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (2m) 8s ago 12m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (112s) 4s ago 15m 40.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (78s) 8s ago 12m 30.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (3m) 4s ago 14m 7612k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:26:30.806 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (3m) 8s ago 12m 7578k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (11m) 4s ago 11m 46.8M 1559M 17.2.7-847-g45317f95 b333581ccdf5 0926f8ab11a8 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (10m) 4s ago 10m 49.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (10m) 4s ago 10m 47.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (9m) 8s ago 9m 46.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (9m) 8s ago 9m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (8m) 8s ago 8m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:26:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (2m) 4s ago 13m 54.8M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:26:30.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 msgr2=0x7fcf480710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 0x7fcf480710b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000b970 tx=0x7fcf50009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:30.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 msgr2=0x7fcf64071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf54002a80 tx=0x7fcf54004600 comp rx=0 tx=0).stop 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 shutdown_connections 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf4806ec00 0x7fcf480710b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf640fcef0 0x7fcf64071df0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 --2- 172.21.15.26:0/1746927421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf640fe0e0 0x7fcf64070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.809+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 >> 172.21.15.26:0/1746927421 conn(0x7fcf640f8530 msgr2=0x7fcf640f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.810+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 shutdown_connections 2024-04-16T07:26:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.810+0000 7fcf6b000700 1 -- 172.21.15.26:0/1746927421 wait complete. 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 -- 172.21.15.26:0/3995287462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 msgr2=0x7f1eb80fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 --2- 172.21.15.26:0/3995287462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb80fe550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4004940 tx=0x7f1ea4009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 -- 172.21.15.26:0/3995287462 shutdown_connections 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 --2- 172.21.15.26:0/3995287462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb80fe550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 --2- 172.21.15.26:0/3995287462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 0x7f1eb80fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.959+0000 7f1ebe200700 1 -- 172.21.15.26:0/3995287462 >> 172.21.15.26:0/3995287462 conn(0x7f1eb80f8530 msgr2=0x7f1eb80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.960+0000 7f1ebe200700 1 -- 172.21.15.26:0/3995287462 shutdown_connections 2024-04-16T07:26:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.960+0000 7f1ebe200700 1 -- 172.21.15.26:0/3995287462 wait complete. 2024-04-16T07:26:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.960+0000 7f1ebe200700 1 Processor -- start 2024-04-16T07:26:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1ebe200700 1 -- start start 2024-04-16T07:26:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1ebe200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 0x7f1eb810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1ebe200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1ebe200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8110850 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1ebe200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8110f90 con 0x7f1eb80fcef0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1eb6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1eb6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49396/0 (socket says 172.21.15.26:49396) 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.961+0000 7f1eb6a00700 1 -- 172.21.15.26:0/2293182579 learned_addr learned my addr 172.21.15.26:0/2293182579 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eb7400700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 0x7f1eb810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eb6a00700 1 -- 172.21.15.26:0/2293182579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 msgr2=0x7f1eb810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eb6a00700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 0x7f1eb810fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eb6a00700 1 -- 172.21.15.26:0/2293182579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4005040 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eb6a00700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b3b0 tx=0x7f1ea4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea400e040 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ea4004290 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8111210 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea401c630 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.962+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb81117d0 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.963+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb805f5f0 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.964+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ea400bbf0 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.965+0000 7f1eaf400700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 0x7f1e9c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.965+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ea408fc70 con 0x7f1eb80fe0e0 2024-04-16T07:26:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.966+0000 7f1eb7400700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 0x7f1e9c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.966+0000 7f1eb7400700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 0x7f1e9c0710c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80fcc30 tx=0x7f1ea800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:30.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:30.970+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1ea4096080 con 0x7f1eb80fe0e0 2024-04-16T07:26:31.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:31.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:31.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:31.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:31.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:31.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:31.062+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:30 smithi104 ceph-mon[81027]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:30 smithi104 ceph-mon[81027]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:31.118 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:30 smithi026 ceph-mon[92215]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:31.118 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:31.118 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:31.118 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:30 smithi026 ceph-mon[92215]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:31.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.294+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1eb8111ae0 con 0x7f1eb80fe0e0 2024-04-16T07:26:31.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.295+0000 7f1eaf400700 1 -- 172.21.15.26:0/2293182579 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f1ea405c5d0 con 0x7f1eb80fe0e0 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:26:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:26:31.296 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:26:31.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.298+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 msgr2=0x7f1e9c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.298+0000 7f1ebe200700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 0x7f1e9c0710c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80fcc30 tx=0x7f1ea800b040 comp rx=0 tx=0).stop 2024-04-16T07:26:31.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.298+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 msgr2=0x7f1eb8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.298+0000 7f1ebe200700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b3b0 tx=0x7f1ea4009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 shutdown_connections 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1e9c06ec10 0x7f1e9c0710c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1eb80fcef0 0x7f1eb810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 --2- 172.21.15.26:0/2293182579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1eb80fe0e0 0x7f1eb8110250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 >> 172.21.15.26:0/2293182579 conn(0x7f1eb80f8530 msgr2=0x7f1eb80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 shutdown_connections 2024-04-16T07:26:31.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.299+0000 7f1ebe200700 1 -- 172.21.15.26:0/2293182579 wait complete. 2024-04-16T07:26:31.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.313+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a00040a0 con 0x55fc3f5009a0 2024-04-16T07:26:31.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.314+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a00040a0 con 0x55fc3f5009a0 2024-04-16T07:26:31.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:31.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:31.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:31.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:31.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:31.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:31.350+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:31.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 -- 172.21.15.26:0/333626441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fbd40 msgr2=0x7fd9600fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 --2- 172.21.15.26:0/333626441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fbd40 0x7fd9600fc150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd950004940 tx=0x7fd950009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:31.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 -- 172.21.15.26:0/333626441 shutdown_connections 2024-04-16T07:26:31.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 --2- 172.21.15.26:0/333626441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fcff0 0x7fd9600fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 --2- 172.21.15.26:0/333626441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fbd40 0x7fd9600fc150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 -- 172.21.15.26:0/333626441 >> 172.21.15.26:0/333626441 conn(0x7fd9600f74c0 msgr2=0x7fd9600f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 -- 172.21.15.26:0/333626441 shutdown_connections 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.452+0000 7fd966800700 1 -- 172.21.15.26:0/333626441 wait complete. 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.453+0000 7fd966800700 1 Processor -- start 2024-04-16T07:26:31.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd966800700 1 -- start start 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd966800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fbd40 0x7fd9600ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd966800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd966800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600fe8c0 con 0x7fd9600fcff0 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd966800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600fea00 con 0x7fd9600fbd40 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd95ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.454+0000 7fd95ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49410/0 (socket says 172.21.15.26:49410) 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95ea00700 1 -- 172.21.15.26:0/1335129358 learned_addr learned my addr 172.21.15.26:0/1335129358 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95f400700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fbd40 0x7fd9600ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95ea00700 1 -- 172.21.15.26:0/1335129358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fbd40 msgr2=0x7fd9600ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95ea00700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fbd40 0x7fd9600ffe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95ea00700 1 -- 172.21.15.26:0/1335129358 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950005040 con 0x7fd9600fcff0 2024-04-16T07:26:31.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.455+0000 7fd95ea00700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd94c002a50 tx=0x7fd94c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:31.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.456+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd94c007cc0 con 0x7fd9600fcff0 2024-04-16T07:26:31.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.456+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd94c00f040 con 0x7fd9600fcff0 2024-04-16T07:26:31.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.456+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9600fece0 con 0x7fd9600fcff0 2024-04-16T07:26:31.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.456+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd94c00da30 con 0x7fd9600fcff0 2024-04-16T07:26:31.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.456+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd960115aa0 con 0x7fd9600fcff0 2024-04-16T07:26:31.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.457+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd960101b80 con 0x7fd9600fcff0 2024-04-16T07:26:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.458+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd94c00db90 con 0x7fd9600fcff0 2024-04-16T07:26:31.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.459+0000 7fd957400700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 0x7fd9440710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.459+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd94c08e5d0 con 0x7fd9600fcff0 2024-04-16T07:26:31.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.460+0000 7fd95f400700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 0x7fd9440710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.461+0000 7fd95f400700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 0x7fd9440710c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd95000b3b0 tx=0x7fd950009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.464+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd94c057380 con 0x7fd9600fcff0 2024-04-16T07:26:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.739+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd960104900 con 0x7fd9600fcff0 2024-04-16T07:26:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.740+0000 7fd957400700 1 -- 172.21.15.26:0/1335129358 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fd94c05aa40 con 0x7fd9600fcff0 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:26:31.740 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:26:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:26:31.742 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 msgr2=0x7fd9440710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 0x7fd9440710c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd95000b3b0 tx=0x7fd950009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 msgr2=0x7fd9600fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd94c002a50 tx=0x7fd94c0046b0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 shutdown_connections 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd94406ec10 0x7fd9440710c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9600fbd40 0x7fd9600ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 --2- 172.21.15.26:0/1335129358 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9600fcff0 0x7fd9600fe330 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.744+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 >> 172.21.15.26:0/1335129358 conn(0x7fd9600f74c0 msgr2=0x7fd960065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.745+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 shutdown_connections 2024-04-16T07:26:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.745+0000 7fd966800700 1 -- 172.21.15.26:0/1335129358 wait complete. 2024-04-16T07:26:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:26:31.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 -- 172.21.15.26:0/2723292413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80fff70 msgr2=0x7f51d8102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 --2- 172.21.15.26:0/2723292413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80fff70 0x7f51d8102390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51c4004940 tx=0x7f51c4009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:31.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 -- 172.21.15.26:0/2723292413 shutdown_connections 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 --2- 172.21.15.26:0/2723292413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80fff70 0x7f51d8102390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 --2- 172.21.15.26:0/2723292413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80662f0 0x7f51d80ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 -- 172.21.15.26:0/2723292413 >> 172.21.15.26:0/2723292413 conn(0x7f51d80f74c0 msgr2=0x7f51d80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 -- 172.21.15.26:0/2723292413 shutdown_connections 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.900+0000 7f51de600700 1 -- 172.21.15.26:0/2723292413 wait complete. 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.901+0000 7f51de600700 1 Processor -- start 2024-04-16T07:26:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.901+0000 7f51de600700 1 -- start start 2024-04-16T07:26:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51de600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51de600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 0x7f51d810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51de600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d810e5a0 con 0x7f51d80662f0 2024-04-16T07:26:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51de600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d810edf0 con 0x7f51d80fff70 2024-04-16T07:26:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51d7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51d7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49418/0 (socket says 172.21.15.26:49418) 2024-04-16T07:26:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51d7400700 1 -- 172.21.15.26:0/32758562 learned_addr learned my addr 172.21.15.26:0/32758562 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.902+0000 7f51d6a00700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 0x7f51d810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51d7400700 1 -- 172.21.15.26:0/32758562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 msgr2=0x7f51d810dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51d7400700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 0x7f51d810dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51d7400700 1 -- 172.21.15.26:0/32758562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c4005040 con 0x7f51d80662f0 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51d6a00700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 0x7f51d810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51d7400700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f51c8002a50 tx=0x7f51c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c8007cc0 con 0x7f51d80662f0 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51c800f040 con 0x7f51d80662f0 2024-04-16T07:26:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c800daf0 con 0x7f51d80662f0 2024-04-16T07:26:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.904+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d810f0d0 con 0x7f51d80662f0 2024-04-16T07:26:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.905+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d810f610 con 0x7f51d80662f0 2024-04-16T07:26:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.906+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d805f5f0 con 0x7f51d80662f0 2024-04-16T07:26:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.908+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51c8009810 con 0x7f51d80662f0 2024-04-16T07:26:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.909+0000 7f51cf400700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 0x7f51bc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.909+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f51c808e5d0 con 0x7f51d80662f0 2024-04-16T07:26:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.910+0000 7f51d6a00700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 0x7f51bc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.911+0000 7f51d6a00700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 0x7f51bc0710c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f51c400b7a0 tx=0x7f51c4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:31.913+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f51c805aeb0 con 0x7f51d80662f0 2024-04-16T07:26:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:31 smithi026 ceph-mon[92215]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:31 smithi026 ceph-mon[92215]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:32.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:31 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2293182579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:32.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:31 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1335129358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 59 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:32.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.062+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:31 smithi104 ceph-mon[81027]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:31 smithi104 ceph-mon[81027]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:31 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2293182579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:31 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1335129358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:26:32.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97517 2024-04-16T07:26:32.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 124, success, outsize: 120 2024-04-16T07:26:32.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.105+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.105+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.105+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.105+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.105+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97614 2024-04-16T07:26:32.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 126, success, outsize: 120 2024-04-16T07:26:32.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.128+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.128+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.128+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.128+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.128+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97714 2024-04-16T07:26:32.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 128, success, outsize: 120 2024-04-16T07:26:32.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.151+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.151+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.151+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.151+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.151+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.157+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d810f9f0 con 0x7f51bc06ec10 2024-04-16T07:26:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.160+0000 7f51cf400700 1 -- 172.21.15.26:0/32758562 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7f51d810f9f0 con 0x7f51bc06ec10 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "6/21 daemons upgraded", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading crash daemons", 2024-04-16T07:26:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:26:32.160 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:26:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 msgr2=0x7f51bc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 0x7f51bc0710c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f51c400b7a0 tx=0x7f51c4009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 msgr2=0x7f51d810da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f51c8002a50 tx=0x7f51c8004600 comp rx=0 tx=0).stop 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 shutdown_connections 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f51bc06ec10 0x7f51bc0710c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d80662f0 0x7f51d810da60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 --2- 172.21.15.26:0/32758562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51d80fff70 0x7f51d810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.162+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 >> 172.21.15.26:0/32758562 conn(0x7f51d80f74c0 msgr2=0x7f51d80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.163+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 shutdown_connections 2024-04-16T07:26:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.163+0000 7f51de600700 1 -- 172.21.15.26:0/32758562 wait complete. 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97822 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 130, success, outsize: 120 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.177+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.177+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.177+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.177+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.177+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97926 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 132, success, outsize: 120 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.196+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.196+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.196+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.196+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.196+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98023 2024-04-16T07:26:32.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 134, success, outsize: 120 2024-04-16T07:26:32.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.212+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.212+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.212+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.212+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.212+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98123 2024-04-16T07:26:32.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 136, success, outsize: 120 2024-04-16T07:26:32.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.238+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.238+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.238+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.238+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.238+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98229 2024-04-16T07:26:32.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 138, success, outsize: 120 2024-04-16T07:26:32.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.263+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.263+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.263+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.263+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.263+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98332 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 140, success, outsize: 120 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.291+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.291+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.291+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.291+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.291+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98429 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 142, success, outsize: 120 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.316+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.316+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.316+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.316+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.316+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98531 2024-04-16T07:26:32.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 144, success, outsize: 120 2024-04-16T07:26:32.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.340+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.340+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.340+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.340+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.341+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 20) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:32.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.350+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:32.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.351+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 24 ==== client_session(renewcaps seq 20) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:32.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:32.351+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 20) v1 from mds.0 2024-04-16T07:26:32.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 -- 172.21.15.26:0/2976257491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 msgr2=0x7f841c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 --2- 172.21.15.26:0/2976257491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c0fd320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f840c004940 tx=0x7f840c009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 -- 172.21.15.26:0/2976257491 shutdown_connections 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 --2- 172.21.15.26:0/2976257491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 0x7f841c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 --2- 172.21.15.26:0/2976257491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c0fd320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 -- 172.21.15.26:0/2976257491 >> 172.21.15.26:0/2976257491 conn(0x7f841c0f8530 msgr2=0x7f841c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 -- 172.21.15.26:0/2976257491 shutdown_connections 2024-04-16T07:26:32.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.352+0000 7f8423000700 1 -- 172.21.15.26:0/2976257491 wait complete. 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.353+0000 7f8423000700 1 Processor -- start 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.353+0000 7f8423000700 1 -- start start 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f8423000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f8423000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 0x7f841c110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f8423000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c110850 con 0x7f841c0fcef0 2024-04-16T07:26:32.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f8423000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c110f90 con 0x7f841c0fe0e0 2024-04-16T07:26:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f841be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f841be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49444/0 (socket says 172.21.15.26:49444) 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.354+0000 7f841be00700 1 -- 172.21.15.26:0/2910210451 learned_addr learned my addr 172.21.15.26:0/2910210451 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f841b400700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 0x7f841c110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f841be00700 1 -- 172.21.15.26:0/2910210451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 msgr2=0x7f841c110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f841be00700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 0x7f841c110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f841be00700 1 -- 172.21.15.26:0/2910210451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c005040 con 0x7f841c0fcef0 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f841be00700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f840c00b970 tx=0x7f840c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.355+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f840c00e040 con 0x7f841c0fcef0 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.356+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f840c004290 con 0x7f841c0fcef0 2024-04-16T07:26:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.356+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c111210 con 0x7f841c0fcef0 2024-04-16T07:26:32.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.356+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c1117d0 con 0x7f841c0fcef0 2024-04-16T07:26:32.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.356+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f840c01c630 con 0x7f841c0fcef0 2024-04-16T07:26:32.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.357+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc0052e0 con 0x7f841c0fcef0 2024-04-16T07:26:32.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.358+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f840c013420 con 0x7f841c0fcef0 2024-04-16T07:26:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.359+0000 7f8413e00700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 0x7f84000710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:26:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.359+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f840c08fb70 con 0x7f841c0fcef0 2024-04-16T07:26:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.360+0000 7f841b400700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 0x7f84000710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:26:32.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.360+0000 7f841b400700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 0x7f84000710c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8408007fd0 tx=0x7f8408009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98655 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.363+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 146, success, outsize: 120 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.363+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.363+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.363+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.363+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.366+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f840c05c4d0 con 0x7f841c0fcef0 2024-04-16T07:26:32.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98758 2024-04-16T07:26:32.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 148, success, outsize: 120 2024-04-16T07:26:32.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.382+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.382+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.382+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.382+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.382+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98855 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 150, success, outsize: 120 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.405+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.405+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.405+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.405+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.405+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98955 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 152, success, outsize: 120 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.431+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.431+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.431+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.431+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.431+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99061 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 154, success, outsize: 120 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.452+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.452+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.452+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.452+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.452+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99164 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 156, success, outsize: 120 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.471+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.471+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.471+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.471+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.471+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99261 2024-04-16T07:26:32.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 158, success, outsize: 120 2024-04-16T07:26:32.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.490+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.490+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.490+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.490+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.490+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99361 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 160, success, outsize: 120 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.508+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.508+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.508+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.508+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.508+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99467 2024-04-16T07:26:32.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 162, success, outsize: 120 2024-04-16T07:26:32.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.528+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.528+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.528+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.528+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.528+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99570 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 164, success, outsize: 120 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.548+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.548+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.548+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.548+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.548+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99667 2024-04-16T07:26:32.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 166, success, outsize: 120 2024-04-16T07:26:32.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.566+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.566+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.566+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.566+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.566+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99767 2024-04-16T07:26:32.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 168, success, outsize: 120 2024-04-16T07:26:32.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.586+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.586+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.586+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.586+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.586+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99873 2024-04-16T07:26:32.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 170, success, outsize: 120 2024-04-16T07:26:32.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.609+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.609+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.609+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.609+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.609+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99976 2024-04-16T07:26:32.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 172, success, outsize: 120 2024-04-16T07:26:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.632+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.632+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.632+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.632+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.632+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100073 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 174, success, outsize: 120 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.653+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.653+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.653+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.653+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.653+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100173 2024-04-16T07:26:32.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 176, success, outsize: 120 2024-04-16T07:26:32.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.675+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.675+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.675+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.675+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.675+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100279 2024-04-16T07:26:32.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 178, success, outsize: 120 2024-04-16T07:26:32.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.698+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:32.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.698+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:32.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.698+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:32.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.698+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:32.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:32.698+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:32.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.715+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f83fc005590 con 0x7f841c0fcef0 2024-04-16T07:26:32.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.716+0000 7f8413e00700 1 -- 172.21.15.26:0/2910210451 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f840c013e60 con 0x7f841c0fcef0 2024-04-16T07:26:32.715 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:26:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 msgr2=0x7f84000710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 0x7f84000710c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8408007fd0 tx=0x7f8408009040 comp rx=0 tx=0).stop 2024-04-16T07:26:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 msgr2=0x7f841c10fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:26:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f840c00b970 tx=0x7f840c009e30 comp rx=0 tx=0).stop 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 shutdown_connections 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f840006ec10 0x7f84000710c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f841c0fcef0 0x7f841c10fd10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 --2- 172.21.15.26:0/2910210451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f841c0fe0e0 0x7f841c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 >> 172.21.15.26:0/2910210451 conn(0x7f841c0f8530 msgr2=0x7f841c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 shutdown_connections 2024-04-16T07:26:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:26:32.721+0000 7f8423000700 1 -- 172.21.15.26:0/2910210451 wait complete. 2024-04-16T07:26:33.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:32 smithi026 ceph-mon[92215]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:33.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:32 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2910210451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:26:33.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:33.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:33.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:33.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:33.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:33.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:33.062+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:33.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:32 smithi104 ceph-mon[81027]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T07:26:33.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:32 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2910210451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:33.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:33.351+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:34.063+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:34.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:26:34.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:26:34.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:34.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:26:34.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:34.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:34.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:34.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:34.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:34.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:34.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:34.351+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:35.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:35.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:35.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:35.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:35.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:35.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:35.063+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi026"}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi026"}]': finished 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]': finished 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T07:26:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi026"}]: dispatch 2024-04-16T07:26:35.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi026"}]': finished 2024-04-16T07:26:35.099 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]: dispatch 2024-04-16T07:26:35.099 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]': finished 2024-04-16T07:26:35.099 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T07:26:35.099 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:35.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:35.351+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:36.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:36.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:36.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:36.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:36.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:36.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:36.063+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:35 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all crash 2024-04-16T07:26:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:35 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T07:26:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:35 smithi104 ceph-mon[81027]: Upgrade: osd.0 is safe to restart 2024-04-16T07:26:36.109 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:35 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all crash 2024-04-16T07:26:36.109 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:35 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T07:26:36.109 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:35 smithi026 ceph-mon[92215]: Upgrade: osd.0 is safe to restart 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:36.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:36.352+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:37.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:37.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:37.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:37.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:37.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:37.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:37.063+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:37.077 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:36 smithi026 ceph-mon[92215]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:37.077 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:37.077 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:26:37.077 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:36 smithi104 ceph-mon[81027]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T07:26:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:26:37.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:37.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:37.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:37.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:37.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:37.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:37.352+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:37.983 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:37 smithi026 ceph-mon[92215]: Upgrade: Updating osd.0 2024-04-16T07:26:37.983 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:37 smithi026 ceph-mon[92215]: Deploying daemon osd.0 on smithi026 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:38.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:38.064+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:37 smithi104 ceph-mon[81027]: Upgrade: Updating osd.0 2024-04-16T07:26:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:37 smithi104 ceph-mon[81027]: Deploying daemon osd.0 on smithi026 2024-04-16T07:26:38.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:38.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:38.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:38.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:38.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:38.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:38.352+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:39.001 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:38 smithi026 ceph-mon[92215]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:39.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:39.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:39.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:39.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:39.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:39.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:39.064+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:39.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:38 smithi104 ceph-mon[81027]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:39.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:39.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:39.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:39.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:39.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:39.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:39.352+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.064+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.065+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.065+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.065+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.065+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:40.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:40.065+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:40.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:40.353+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:41.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:41.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:41.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:41.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:41.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:41.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:41.065+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:41.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:40 smithi026 ceph-mon[92215]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:40 smithi104 ceph-mon[81027]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:41.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:41.353+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:41.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:41 smithi026 systemd[1]: Stopping Ceph osd.0 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:26:42.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:42.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:42.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:42.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:42.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:42.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:42.066+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:42.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:41 smithi026 ceph-mon[92215]: osd.0 marked itself down and dead 2024-04-16T07:26:42.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[46938]: 2024-04-16T07:26:41.771+0000 7f8f9f400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:26:42.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[46938]: 2024-04-16T07:26:41.771+0000 7f8f9f400700 -1 osd.0 50 *** Got signal Terminated *** 2024-04-16T07:26:42.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[46938]: 2024-04-16T07:26:41.771+0000 7f8f9f400700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:26:42.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:41 smithi104 ceph-mon[81027]: osd.0 marked itself down and dead 2024-04-16T07:26:42.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:42.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:42.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:42.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:42.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:42.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:42.353+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:42.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:42 smithi026 podman[100736]: 2024-04-16 07:26:42.287604103 +0000 UTC m=+0.652094625 container died 0926f8ab11a877c158e59bcf03ca418851ba568a3a75bd17a66fa17c4848d895 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-16T07:26:42.848 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:42 smithi026 podman[100736]: 2024-04-16 07:26:42.714072432 +0000 UTC m=+1.078562991 container cleanup 0926f8ab11a877c158e59bcf03ca418851ba568a3a75bd17a66fa17c4848d895 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-16T07:26:42.848 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:42 smithi026 bash[100736]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0 2024-04-16T07:26:43.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:43.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:43.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:43.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:43.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:43.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:43.066+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:43.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:42 smithi026 ceph-mon[92215]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:43.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:42 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:26:43.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:42 smithi026 ceph-mon[92215]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T07:26:43.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:43 smithi026 podman[100755]: 2024-04-16 07:26:43.006956672 +0000 UTC m=+0.707689948 container remove 0926f8ab11a877c158e59bcf03ca418851ba568a3a75bd17a66fa17c4848d895 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T07:26:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:42 smithi104 ceph-mon[81027]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:43.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:42 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:26:43.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:42 smithi104 ceph-mon[81027]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:43.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:43.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:44.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:44.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 21) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.066+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:44.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.067+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 26 ==== client_session(renewcaps seq 21) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:26:44.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:44.067+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 21) v1 from mds.0 2024-04-16T07:26:44.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:43 smithi026 ceph-mon[92215]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T07:26:44.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:43 smithi026 podman[100839]: 2024-04-16 07:26:43.762435013 +0000 UTC m=+0.036055748 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:26:44.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:43 smithi026 podman[100839]: 2024-04-16 07:26:43.866095169 +0000 UTC m=+0.139715940 container create 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-16T07:26:44.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:44 smithi026 podman[100839]: 2024-04-16 07:26:44.162347238 +0000 UTC m=+0.435968062 container init 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-16T07:26:44.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:43 smithi104 ceph-mon[81027]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:44.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:44.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:44.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:44 smithi026 podman[100839]: 2024-04-16 07:26:44.174792912 +0000 UTC m=+0.448413717 container start 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T07:26:44.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:44 smithi026 podman[100839]: 2024-04-16 07:26:44.301934345 +0000 UTC m=+0.575555117 container attach 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-04-16T07:26:45.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:44 smithi026 ceph-mon[92215]: pgmap v62: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:45.011 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:44 smithi026 podman[100839]: 2024-04-16 07:26:44.658820924 +0000 UTC m=+0.932441702 container died 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:45.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:45.066+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:45.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:44 smithi104 ceph-mon[81027]: pgmap v62: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:26:45.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:45.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:45.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:45.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:45 smithi026 podman[100865]: 2024-04-16 07:26:45.389255448 +0000 UTC m=+0.712640723 container remove 666bc69c0ebd1908cee739267a6af5bfe6b5e4455211c4a56633f45c59f12949 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-16T07:26:45.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:45 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-16T07:26:45.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:45 smithi026 systemd[1]: Stopped Ceph osd.0 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:26:45.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:45 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.0.service: Consumed 12.872s CPU time. 2024-04-16T07:26:46.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:46.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:46.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:46.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:46.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:46.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:46.067+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:46.220 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:45 smithi026 systemd[1]: Starting Ceph osd.0 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:26:46.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:45 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T07:26:46.234 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:45 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T07:26:46.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:46.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:46.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:46.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:46.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:46.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:46.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:46.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:46 smithi026 podman[100965]: 2024-04-16 07:26:46.127349235 +0000 UTC m=+0.036696437 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:26:46.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:46 smithi026 podman[100965]: 2024-04-16 07:26:46.223081784 +0000 UTC m=+0.132429023 container create 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-16T07:26:46.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:46 smithi026 podman[100965]: 2024-04-16 07:26:46.610182637 +0000 UTC m=+0.519529896 container init 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-16T07:26:46.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:46 smithi026 podman[100965]: 2024-04-16 07:26:46.623357202 +0000 UTC m=+0.532704423 container start 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-16T07:26:46.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:46 smithi026 podman[100965]: 2024-04-16 07:26:46.708892311 +0000 UTC m=+0.618239559 container attach 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 87 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:47.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.067+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:47.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:46 smithi026 ceph-mon[92215]: pgmap v63: 65 pgs: 2 active+undersized+degraded, 9 active+undersized, 46 active+clean, 8 stale+active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T07:26:47.262 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:46 smithi104 ceph-mon[81027]: pgmap v63: 65 pgs: 2 active+undersized+degraded, 9 active+undersized, 46 active+clean, 8 stale+active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:47.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:47.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:47.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101080 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 180, success, outsize: 120 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.426+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.426+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.426+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.426+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:26:47.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:47.426+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T07:26:48.003 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 bash[100965]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate[100979]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T07:26:48.004 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:47 smithi026 podman[101113]: 2024-04-16 07:26:47.69483026 +0000 UTC m=+0.041078734 container died 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:26:48.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:48.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:48.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:48.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:48.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:48.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:48.067+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:48.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:48.354+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:48.760 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:48 smithi026 podman[101113]: 2024-04-16 07:26:48.446212729 +0000 UTC m=+0.792461239 container remove 2c6ef0b6d2188ea3680544b121ad99c72339193a87ecf7694a15366b6b6bfbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:49.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:49.067+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:48 smithi104 ceph-mon[81027]: pgmap v64: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:49.135 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:48 smithi026 ceph-mon[92215]: pgmap v64: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:49.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:48 smithi026 podman[101175]: 2024-04-16 07:26:48.711975283 +0000 UTC m=+0.035258673 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:26:49.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:48 smithi026 podman[101175]: 2024-04-16 07:26:48.963800624 +0000 UTC m=+0.287084036 container create 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T07:26:49.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:49.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:49.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:49.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:49.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:49.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:49.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:49.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 podman[101175]: 2024-04-16 07:26:49.217207729 +0000 UTC m=+0.540491185 container init 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T07:26:49.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 podman[101175]: 2024-04-16 07:26:49.22853724 +0000 UTC m=+0.551820685 container start 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-16T07:26:49.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 ceph-osd[101193]: -- 172.21.15.26:0/591635254 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558730ce3680 con 0x558730cacc00 2024-04-16T07:26:49.512 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 bash[101175]: 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea 2024-04-16T07:26:49.512 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 systemd[1]: Started Ceph osd.0 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.067+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.067+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.067+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.067+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.068+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:50.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:50.068+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:50.260 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:49 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:26:49.813+0000 7fd6c08ee7c0 -1 Falling back to public interface 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.354+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.354+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:50.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:50.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: pgmap v65: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: pgmap v65: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:26:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:26:51.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:51.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:51.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:51.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:51.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:51.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:51.068+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:51.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:51.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:51.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:51.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:51.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:51.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:51.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:52.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:52.068+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 21) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:52.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:52.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 25 ==== client_session(renewcaps seq 21) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:52.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:52.355+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 21) v1 from mds.0 2024-04-16T07:26:52.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:52 smithi026 ceph-mon[92215]: pgmap v66: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:52 smithi104 ceph-mon[81027]: pgmap v66: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:53.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:53.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:53.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:53.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:53.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:53.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:53.068+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:53.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:53.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:53.505 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:53 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:26:53.459+0000 7fd6c08ee7c0 -1 osd.0 50 log_to_monitors true 2024-04-16T07:26:54.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:54.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:54.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:54.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:54.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:54.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:54.068+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:54.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:54.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:54.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:54.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:54.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:54.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:54.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:54.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:54 smithi026 ceph-mon[92215]: pgmap v67: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:54.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:54 smithi026 ceph-mon[92215]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T07:26:54.756 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:54 smithi026 ceph-mon[92215]: Health check update: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T07:26:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:54 smithi104 ceph-mon[81027]: pgmap v67: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:54 smithi104 ceph-mon[81027]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T07:26:54.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:54 smithi104 ceph-mon[81027]: Health check update: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.068+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.068+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.068+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.068+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.069+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:55.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:55.069+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:55.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:55.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:55.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:55.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:55.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:55.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:55.355+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:55.648 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:26:55 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:26:55.388+0000 7fd6aac00700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:26:55.649 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:55 smithi026 ceph-mon[92215]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T07:26:55.649 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:55 smithi026 ceph-mon[92215]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T07:26:55.649 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:55 smithi026 ceph-mon[92215]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:26:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:55 smithi104 ceph-mon[81027]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T07:26:55.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:55 smithi104 ceph-mon[81027]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T07:26:55.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:55 smithi104 ceph-mon[81027]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:56.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:56.069+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:56.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:56.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:56.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:56.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:56.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:56.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:56.356+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:56.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:56 smithi026 ceph-mon[92215]: pgmap v69: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:56.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:56 smithi026 ceph-mon[92215]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' 2024-04-16T07:26:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:56 smithi104 ceph-mon[81027]: pgmap v69: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:56.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:56 smithi104 ceph-mon[81027]: from='osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947]' entity='osd.0' 2024-04-16T07:26:57.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:57.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:57.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:57.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:57.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:57.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:57.069+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:57.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:57.357+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947] boot 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: osdmap e54: 6 total, 6 up, 6 in 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:26:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: osd.0 [v2:172.21.15.26:6802/214439947,v1:172.21.15.26:6803/214439947] boot 2024-04-16T07:26:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: osdmap e54: 6 total, 6 up, 6 in 2024-04-16T07:26:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T07:26:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:57.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:58.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:58.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:58.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:58.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:58.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:58.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:58.069+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:58.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:58.357+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:58 smithi104 ceph-mon[81027]: pgmap v71: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:58.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:58 smithi104 ceph-mon[81027]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T07:26:58.664 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:58 smithi026 ceph-mon[92215]: pgmap v71: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:26:58.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:58 smithi026 ceph-mon[92215]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.069+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.069+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.069+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.069+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.070+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:26:59.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:26:59.070+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:59.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:26:59.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:26:59.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:26:59.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:26:59.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:26:59.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:26:59.357+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:26:59.910 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:59 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:26:59.911 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:26:59 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:00.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:00.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:00.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:00.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:00.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:00.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:00.070+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:00.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:59 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:00.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:26:59 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:00.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.356+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:00.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.356+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:00.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.356+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:00.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.356+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:00.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.357+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:00.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:00.357+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:01.007 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:00 smithi026 ceph-mon[92215]: pgmap v73: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:01.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:01.070+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:00 smithi104 ceph-mon[81027]: pgmap v73: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:01.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:01.358+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:02.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:01 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded) 2024-04-16T07:27:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:01 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 88 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:02.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.071+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:02.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:01 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded) 2024-04-16T07:27:02.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:01 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:27:02.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:02.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:02.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:02.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:02.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:02.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:02.358+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:02.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103145 2024-04-16T07:27:02.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 182, success, outsize: 120 2024-04-16T07:27:02.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.848+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.848+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.848+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.848+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.848+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103239 2024-04-16T07:27:02.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 184, success, outsize: 120 2024-04-16T07:27:02.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.873+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.873+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.873+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.873+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.873+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103336 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 186, success, outsize: 120 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.890+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.890+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.890+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.890+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.890+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 -- 172.21.15.26:0/4095920504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 msgr2=0x7fe5580ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 --2- 172.21.15.26:0/4095920504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe5580ff9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe54c004940 tx=0x7fe54c009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 -- 172.21.15.26:0/4095920504 shutdown_connections 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 --2- 172.21.15.26:0/4095920504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5580fff20 0x7fe558102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 --2- 172.21.15.26:0/4095920504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe5580ff9e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 -- 172.21.15.26:0/4095920504 >> 172.21.15.26:0/4095920504 conn(0x7fe5580f7450 msgr2=0x7fe5580f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 -- 172.21.15.26:0/4095920504 shutdown_connections 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.901+0000 7fe560800700 1 -- 172.21.15.26:0/4095920504 wait complete. 2024-04-16T07:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.902+0000 7fe560800700 1 Processor -- start 2024-04-16T07:27:02.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.902+0000 7fe560800700 1 -- start start 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe560800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe560800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5580fff20 0x7fe55810e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe560800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55810e8d0 con 0x7fe558066f60 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe560800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581131e0 con 0x7fe5580fff20 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe55de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe55de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37694/0 (socket says 172.21.15.26:37694) 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe55de00700 1 -- 172.21.15.26:0/1811073712 learned_addr learned my addr 172.21.15.26:0/1811073712 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.903+0000 7fe55de00700 1 -- 172.21.15.26:0/1811073712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5580fff20 msgr2=0x7fe55810e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe55de00700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5580fff20 0x7fe55810e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe55de00700 1 -- 172.21.15.26:0/1811073712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c005040 con 0x7fe558066f60 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe55de00700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00b970 tx=0x7fe54c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54c00e040 con 0x7fe558066f60 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe54c00bbf0 con 0x7fe558066f60 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54c01c630 con 0x7fe558066f60 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.904+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5581133e0 con 0x7fe558066f60 2024-04-16T07:27:02.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.905+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe558113910 con 0x7fe558066f60 2024-04-16T07:27:02.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.906+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55804c710 con 0x7fe558066f60 2024-04-16T07:27:02.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.907+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe54c00bd60 con 0x7fe558066f60 2024-04-16T07:27:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.908+0000 7fe552000700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 0x7fe5400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.908+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe54c08fca0 con 0x7fe558066f60 2024-04-16T07:27:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.909+0000 7fe55d400700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 0x7fe5400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:02.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.910+0000 7fe55d400700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 0x7fe5400710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe548006fd0 tx=0x7fe548006db0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103456 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 188, success, outsize: 120 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.912+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.912+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.912+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.912+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.912+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:02.914+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe54c058ad0 con 0x7fe558066f60 2024-04-16T07:27:02.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103555 2024-04-16T07:27:02.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 190, success, outsize: 120 2024-04-16T07:27:02.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.931+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.931+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.931+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.931+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.931+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103648 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 192, success, outsize: 120 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.947+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.947+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.947+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.947+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.947+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103744 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 194, success, outsize: 120 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.964+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.964+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.964+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.964+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.964+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103846 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 196, success, outsize: 120 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.983+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.983+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.983+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.983+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:02.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:02.983+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103945 2024-04-16T07:27:03.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 198, success, outsize: 120 2024-04-16T07:27:03.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.001+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.001+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.001+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.001+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.001+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:02 smithi026 ceph-mon[92215]: pgmap v74: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:03.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104038 2024-04-16T07:27:03.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 200, success, outsize: 120 2024-04-16T07:27:03.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.019+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.019+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.019+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.019+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.019+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104134 2024-04-16T07:27:03.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 202, success, outsize: 120 2024-04-16T07:27:03.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.036+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.036+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.036+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.036+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.036+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104236 2024-04-16T07:27:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 204, success, outsize: 120 2024-04-16T07:27:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.056+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.056+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.056+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.056+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.056+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 100 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:03.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.071+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:03.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104335 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 206, success, outsize: 120 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.074+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.074+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.074+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.074+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.074+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:02 smithi104 ceph-mon[81027]: pgmap v74: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104428 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 208, success, outsize: 120 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.090+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.090+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.090+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.090+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.090+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104524 2024-04-16T07:27:03.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 210, success, outsize: 120 2024-04-16T07:27:03.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.108+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.108+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.108+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.108+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.108+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104626 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 212, success, outsize: 120 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.129+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.129+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.129+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.129+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.129+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104725 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 214, success, outsize: 120 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.149+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.149+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.149+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.149+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.150+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104818 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 216, success, outsize: 120 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.167+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.167+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.167+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.167+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.167+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.171+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe558101b90 con 0x7fe54006ec00 2024-04-16T07:27:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.172+0000 7fe552000700 1 -- 172.21.15.26:0/1811073712 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe558101b90 con 0x7fe54006ec00 2024-04-16T07:27:03.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.176+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 msgr2=0x7fe5400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.176+0000 7fe560800700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 0x7fe5400710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe548006fd0 tx=0x7fe548006db0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.176+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 msgr2=0x7fe55810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.176+0000 7fe560800700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00b970 tx=0x7fe54c009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 shutdown_connections 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe54006ec00 0x7fe5400710b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe558066f60 0x7fe55810dcb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 --2- 172.21.15.26:0/1811073712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5580fff20 0x7fe55810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 >> 172.21.15.26:0/1811073712 conn(0x7fe5580f7450 msgr2=0x7fe5580f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:03.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 shutdown_connections 2024-04-16T07:27:03.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.177+0000 7fe560800700 1 -- 172.21.15.26:0/1811073712 wait complete. 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104916 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 218, success, outsize: 120 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.187+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.187+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.187+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.187+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.187+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.194 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:27:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105021 2024-04-16T07:27:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 220, success, outsize: 120 2024-04-16T07:27:03.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.208+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.208+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.208+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.208+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.209+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105120 2024-04-16T07:27:03.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 222, success, outsize: 120 2024-04-16T07:27:03.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.226+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.226+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.226+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.226+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.226+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105213 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 224, success, outsize: 120 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.244+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.244+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.244+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.244+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.244+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105309 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 226, success, outsize: 120 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.263+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.263+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.263+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.263+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.263+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105411 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 228, success, outsize: 120 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.282+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.282+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.282+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.282+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.282+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105510 2024-04-16T07:27:03.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 230, success, outsize: 120 2024-04-16T07:27:03.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.300+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.300+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.300+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.301+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.301+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105605 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 232, success, outsize: 120 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.321+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.321+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.321+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.321+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.321+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.337+0000 7f5a79e00700 1 -- 172.21.15.26:0/4155668992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 msgr2=0x7f5a740fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.337+0000 7f5a79e00700 1 --2- 172.21.15.26:0/4155668992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a740fd250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5a64004940 tx=0x7f5a64009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 -- 172.21.15.26:0/4155668992 shutdown_connections 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 --2- 172.21.15.26:0/4155668992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a740fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 --2- 172.21.15.26:0/4155668992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a740fd250 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 -- 172.21.15.26:0/4155668992 >> 172.21.15.26:0/4155668992 conn(0x7f5a740f8530 msgr2=0x7f5a740fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 -- 172.21.15.26:0/4155668992 shutdown_connections 2024-04-16T07:27:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.338+0000 7f5a79e00700 1 -- 172.21.15.26:0/4155668992 wait complete. 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 Processor -- start 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 -- start start 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a74110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74110880 con 0x7f5a740fce40 2024-04-16T07:27:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.339+0000 7f5a79e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74110fc0 con 0x7f5a740fe040 2024-04-16T07:27:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37704/0 (socket says 172.21.15.26:37704) 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 -- 172.21.15.26:0/1269649000 learned_addr learned my addr 172.21.15.26:0/1269649000 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72000700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a74110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 -- 172.21.15.26:0/1269649000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 msgr2=0x7f5a74110280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a74110280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72a00700 1 -- 172.21.15.26:0/1269649000 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64005040 con 0x7f5a740fce40 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.340+0000 7f5a72000700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a74110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a72a00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400b970 tx=0x7f5a64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6400e040 con 0x7f5a740fce40 2024-04-16T07:27:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a64004290 con 0x7f5a740fce40 2024-04-16T07:27:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a74111240 con 0x7f5a740fce40 2024-04-16T07:27:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6401c630 con 0x7f5a740fce40 2024-04-16T07:27:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.341+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74111800 con 0x7f5a740fce40 2024-04-16T07:27:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.342+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a741019d0 con 0x7f5a740fce40 2024-04-16T07:27:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.344+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5a6400bbf0 con 0x7f5a740fce40 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105720 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 234, success, outsize: 120 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.344+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.344+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.344+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.344+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.344+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.345+0000 7f5a6aa00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 0x7f5a580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.345+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a6408fba0 con 0x7f5a740fce40 2024-04-16T07:27:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.345+0000 7f5a72000700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 0x7f5a580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.346+0000 7f5a72000700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 0x7f5a580710c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a60007fd0 tx=0x7f5a60009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:03.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.350+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5a6405c500 con 0x7f5a740fce40 2024-04-16T07:27:03.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:03.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:03.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:03.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:03.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:03.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:03.358+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105822 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 236, success, outsize: 120 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.363+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.363+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.363+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.363+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:03.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:03.363+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.611+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a74111b30 con 0x7f5a5806ec10 2024-04-16T07:27:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.614+0000 7f5a6aa00700 1 -- 172.21.15.26:0/1269649000 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5a74111b30 con 0x7f5a5806ec10 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 msgr2=0x7f5a580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 0x7f5a580710c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a60007fd0 tx=0x7f5a60009040 comp rx=0 tx=0).stop 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 msgr2=0x7f5a7410fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400b970 tx=0x7f5a64009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 shutdown_connections 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5a5806ec10 0x7f5a580710c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a740fce40 0x7f5a7410fd40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 --2- 172.21.15.26:0/1269649000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a740fe040 0x7f5a74110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 >> 172.21.15.26:0/1269649000 conn(0x7f5a740f8530 msgr2=0x7f5a740f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 shutdown_connections 2024-04-16T07:27:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.618+0000 7f5a79e00700 1 -- 172.21.15.26:0/1269649000 wait complete. 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 -- 172.21.15.26:0/4294650570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 msgr2=0x7f9b100fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 --2- 172.21.15.26:0/4294650570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b100fe350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9afc004940 tx=0x7f9afc009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 -- 172.21.15.26:0/4294650570 shutdown_connections 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 --2- 172.21.15.26:0/4294650570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b100fe350 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 --2- 172.21.15.26:0/4294650570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 0x7f9b100fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.778+0000 7f9b14c00700 1 -- 172.21.15.26:0/4294650570 >> 172.21.15.26:0/4294650570 conn(0x7f9b100f83d0 msgr2=0x7f9b100fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.779+0000 7f9b14c00700 1 -- 172.21.15.26:0/4294650570 shutdown_connections 2024-04-16T07:27:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.779+0000 7f9b14c00700 1 -- 172.21.15.26:0/4294650570 wait complete. 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 Processor -- start 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 -- start start 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 0x7f9b1010d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1010e3d0 con 0x7f9b100fdee0 2024-04-16T07:27:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b14c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1010ebd0 con 0x7f9b100fcc30 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37734/0 (socket says 172.21.15.26:37734) 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 -- 172.21.15.26:0/3468380498 learned_addr learned my addr 172.21.15.26:0/3468380498 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0f400700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 0x7f9b1010d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 -- 172.21.15.26:0/3468380498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 msgr2=0x7f9b1010d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 0x7f9b1010d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 -- 172.21.15.26:0/3468380498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc005040 con 0x7f9b100fdee0 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.781+0000 7f9b0ea00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00b3b0 tx=0x7f9afc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.782+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9afc00e040 con 0x7f9b100fdee0 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.782+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9afc004290 con 0x7f9b100fdee0 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.782+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9afc01c630 con 0x7f9b100fdee0 2024-04-16T07:27:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.782+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1010ee50 con 0x7f9b100fdee0 2024-04-16T07:27:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.782+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1010f390 con 0x7f9b100fdee0 2024-04-16T07:27:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.784+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1005f5f0 con 0x7f9b100fdee0 2024-04-16T07:27:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.784+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9afc00bbf0 con 0x7f9b100fdee0 2024-04-16T07:27:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.786+0000 7f9b07400700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 0x7f9af40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.786+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9afc08fbb0 con 0x7f9b100fdee0 2024-04-16T07:27:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.786+0000 7f9b0f400700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 0x7f9af40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.787+0000 7f9b0f400700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 0x7f9af40710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b00004fa0 tx=0x7f9b0000b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:03.793+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9afc096080 con 0x7f9b100fdee0 2024-04-16T07:27:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.058+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b1010f710 con 0x7f9af406ec10 2024-04-16T07:27:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.070+0000 7f9b07400700 1 -- 172.21.15.26:0/3468380498 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9b1010f710 con 0x7f9af406ec10 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (4m) 7s ago 14m 14.3M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (53s) 7s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (46s) 41s ago 13m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (3m) 7s ago 14m 69.9M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (7m) 7s ago 7m 13.4M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (7m) 7s ago 7m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (7m) 41s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (7m) 41s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (3m) 7s ago 16m 499M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (2m) 41s ago 13m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (2m) 7s ago 16m 43.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:27:04.070 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (111s) 41s ago 12m 30.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (4m) 7s ago 14m 7780k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (3m) 41s ago 13m 7578k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (15s) 7s ago 12m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (11m) 7s ago 11m 49.4M 1559M 17.2.7-847-g45317f95 b333581ccdf5 99152f5e9776 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (10m) 7s ago 10m 47.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (10m) 41s ago 10m 46.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (9m) 41s ago 9m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (9m) 41s ago 9m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:27:04.071 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (2m) 7s ago 13m 56.0M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 22) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 27 ==== client_session(renewcaps seq 22) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:27:04.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:04.071+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 22) v1 from mds.0 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.073+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 msgr2=0x7f9af40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.073+0000 7f9b14c00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 0x7f9af40710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b00004fa0 tx=0x7f9b0000b040 comp rx=0 tx=0).stop 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.073+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 msgr2=0x7f9b1010ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.073+0000 7f9b14c00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00b3b0 tx=0x7f9afc009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.073+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 shutdown_connections 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9af406ec10 0x7f9af40710c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b100fcc30 0x7f9b1010d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 --2- 172.21.15.26:0/3468380498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b100fdee0 0x7f9b1010ddd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 >> 172.21.15.26:0/3468380498 conn(0x7f9b100f83d0 msgr2=0x7f9b100f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 shutdown_connections 2024-04-16T07:27:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.074+0000 7f9b14c00700 1 -- 172.21.15.26:0/3468380498 wait complete. 2024-04-16T07:27:04.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 -- 172.21.15.26:0/376573835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fd710 msgr2=0x7f1a1c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 --2- 172.21.15.26:0/376573835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c0fe070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a08004940 tx=0x7f1a08009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 -- 172.21.15.26:0/376573835 shutdown_connections 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 --2- 172.21.15.26:0/376573835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c0fe070 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 --2- 172.21.15.26:0/376573835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 -- 172.21.15.26:0/376573835 >> 172.21.15.26:0/376573835 conn(0x7f1a1c0f8540 msgr2=0x7f1a1c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 -- 172.21.15.26:0/376573835 shutdown_connections 2024-04-16T07:27:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.228+0000 7f1a22400700 1 -- 172.21.15.26:0/376573835 wait complete. 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.229+0000 7f1a22400700 1 Processor -- start 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a22400700 1 -- start start 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a22400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a22400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c1102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a22400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c1108a0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a22400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c110fe0 con 0x7f1a1c0fd710 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a1b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a1b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37740/0 (socket says 172.21.15.26:37740) 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.230+0000 7f1a1b400700 1 -- 172.21.15.26:0/1269838113 learned_addr learned my addr 172.21.15.26:0/1269838113 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a1aa00700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c1102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a1b400700 1 -- 172.21.15.26:0/1269838113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fd710 msgr2=0x7f1a1c1102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a1b400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c1102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a1b400700 1 -- 172.21.15.26:0/1269838113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a08005040 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a1b400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00c8b0 tx=0x7f1a0c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.231+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0c00a760 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.232+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a0c00ad90 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.232+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c1112c0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.232+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0c0113a0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.232+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c111880 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.233+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c101a60 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.234+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a0c012400 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.235+0000 7f1a13400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 0x7f1a000710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.236+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1a0c08e7a0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.236+0000 7f1a1aa00700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 0x7f1a000710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.237+0000 7f1a1aa00700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 0x7f1a000710b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800b970 tx=0x7f1a08009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.241+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1a0c05b1c0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:04.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:04.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:04.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:04.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:04.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:04.359+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.591+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a1c05f5f0 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.593+0000 7f1a13400700 1 -- 172.21.15.26:0/1269838113 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f1a0c05ad50 con 0x7f1a1c0fcdc0 2024-04-16T07:27:04.592 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:27:04.592 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:27:04.592 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:27:04.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 5, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 9, 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:27:04.593 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:27:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 msgr2=0x7f1a000710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 0x7f1a000710b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800b970 tx=0x7f1a08009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 msgr2=0x7f1a1c10fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00c8b0 tx=0x7f1a0c004470 comp rx=0 tx=0).stop 2024-04-16T07:27:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 shutdown_connections 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1a0006ec00 0x7f1a000710b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a1c0fcdc0 0x7f1a1c10fd60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 --2- 172.21.15.26:0/1269838113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a1c0fd710 0x7f1a1c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 >> 172.21.15.26:0/1269838113 conn(0x7f1a1c0f8540 msgr2=0x7f1a1c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 shutdown_connections 2024-04-16T07:27:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.596+0000 7f1a22400700 1 -- 172.21.15.26:0/1269838113 wait complete. 2024-04-16T07:27:04.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 -- 172.21.15.26:0/2409403449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 msgr2=0x7fee80100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 --2- 172.21.15.26:0/2409403449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80100790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fee6c004940 tx=0x7fee6c009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 -- 172.21.15.26:0/2409403449 shutdown_connections 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 --2- 172.21.15.26:0/2409403449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80100790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 --2- 172.21.15.26:0/2409403449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee800ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.744+0000 7fee86a00700 1 -- 172.21.15.26:0/2409403449 >> 172.21.15.26:0/2409403449 conn(0x7fee800fa950 msgr2=0x7fee800fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.745+0000 7fee86a00700 1 -- 172.21.15.26:0/2409403449 shutdown_connections 2024-04-16T07:27:04.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.745+0000 7fee86a00700 1 -- 172.21.15.26:0/2409403449 wait complete. 2024-04-16T07:27:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 Processor -- start 2024-04-16T07:27:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 -- start start 2024-04-16T07:27:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee8010fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80110760 con 0x7fee80100320 2024-04-16T07:27:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee86a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80110fb0 con 0x7fee800ff070 2024-04-16T07:27:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee7f400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37758/0 (socket says 172.21.15.26:37758) 2024-04-16T07:27:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.746+0000 7fee7f400700 1 -- 172.21.15.26:0/1816421946 learned_addr learned my addr 172.21.15.26:0/1816421946 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7fe00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee8010fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7f400700 1 -- 172.21.15.26:0/1816421946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 msgr2=0x7fee8010fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7f400700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee8010fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7f400700 1 -- 172.21.15.26:0/1816421946 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c005040 con 0x7fee80100320 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7fe00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee8010fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee7f400700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00b3b0 tx=0x7fee6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee6c00e040 con 0x7fee80100320 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee6c004290 con 0x7fee80100320 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee6c01c630 con 0x7fee80100320 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.747+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee80111230 con 0x7fee80100320 2024-04-16T07:27:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.748+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee80111770 con 0x7fee80100320 2024-04-16T07:27:04.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.749+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8005f5f0 con 0x7fee80100320 2024-04-16T07:27:04.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.750+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee6c00bbf0 con 0x7fee80100320 2024-04-16T07:27:04.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.751+0000 7fee77e00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 0x7fee640710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:04.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.752+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fee6c08fc70 con 0x7fee80100320 2024-04-16T07:27:04.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.752+0000 7fee7fe00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 0x7fee640710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:04.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.753+0000 7fee7fe00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 0x7fee640710c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fee70007fd0 tx=0x7fee70009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:04.756+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fee6c096080 con 0x7fee80100320 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: pgmap v75: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1269838113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T07:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.034+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fee80111b40 con 0x7fee80100320 2024-04-16T07:27:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.035+0000 7fee77e00700 1 -- 172.21.15.26:0/1816421946 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fee6c05c5d0 con 0x7fee80100320 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:27:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:27:05.036 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:05.037 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 msgr2=0x7fee640710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 0x7fee640710c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fee70007fd0 tx=0x7fee70009040 comp rx=0 tx=0).stop 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 msgr2=0x7fee80110160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00b3b0 tx=0x7fee6c009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 shutdown_connections 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fee6406ec10 0x7fee640710c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee800ff070 0x7fee8010fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 --2- 172.21.15.26:0/1816421946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee80100320 0x7fee80110160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 >> 172.21.15.26:0/1816421946 conn(0x7fee800fa950 msgr2=0x7fee800fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 shutdown_connections 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.038+0000 7fee86a00700 1 -- 172.21.15.26:0/1816421946 wait complete. 2024-04-16T07:27:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:27:05.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:05.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:05.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:05.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:05.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:05.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:05.071+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: pgmap v75: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1269838113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:05.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:05.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.195+0000 7fd142200700 1 -- 172.21.15.26:0/459442150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c066c80 msgr2=0x7fd13c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.195+0000 7fd142200700 1 --2- 172.21.15.26:0/459442150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c066c80 0x7fd13c101ca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd130004940 tx=0x7fd130009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 -- 172.21.15.26:0/459442150 shutdown_connections 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 --2- 172.21.15.26:0/459442150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c1021e0 0x7fd13c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 --2- 172.21.15.26:0/459442150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c066c80 0x7fd13c101ca0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 -- 172.21.15.26:0/459442150 >> 172.21.15.26:0/459442150 conn(0x7fd13c0f72b0 msgr2=0x7fd13c0f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 -- 172.21.15.26:0/459442150 shutdown_connections 2024-04-16T07:27:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.196+0000 7fd142200700 1 -- 172.21.15.26:0/459442150 wait complete. 2024-04-16T07:27:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.197+0000 7fd142200700 1 Processor -- start 2024-04-16T07:27:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.197+0000 7fd142200700 1 -- start start 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd142200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 0x7fd13c10faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd142200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd142200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c110630 con 0x7fd13c1021e0 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd142200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c110e30 con 0x7fd13c066c80 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37768/0 (socket says 172.21.15.26:37768) 2024-04-16T07:27:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 -- 172.21.15.26:0/578785810 learned_addr learned my addr 172.21.15.26:0/578785810 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd140e00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 0x7fd13c10faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 -- 172.21.15.26:0/578785810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 msgr2=0x7fd13c10faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 0x7fd13c10faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd13be00700 1 -- 172.21.15.26:0/578785810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130005040 con 0x7fd13c1021e0 2024-04-16T07:27:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.198+0000 7fd140e00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 0x7fd13c10faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd13be00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c830 tx=0x7fd128004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12800a760 con 0x7fd13c1021e0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd12800ad90 con 0x7fd13c1021e0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1280113a0 con 0x7fd13c1021e0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c111110 con 0x7fd13c1021e0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.199+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c111650 con 0x7fd13c1021e0 2024-04-16T07:27:05.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.200+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c0faf50 con 0x7fd13c1021e0 2024-04-16T07:27:05.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.202+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd128012400 con 0x7fd13c1021e0 2024-04-16T07:27:05.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.203+0000 7fd138c00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 0x7fd1240710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.203+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd12808e7d0 con 0x7fd13c1021e0 2024-04-16T07:27:05.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.204+0000 7fd140e00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 0x7fd1240710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.204+0000 7fd140e00700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 0x7fd1240710b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd13000b7a0 tx=0x7fd130009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.207+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd12805b1f0 con 0x7fd13c1021e0 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:05.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:05.359+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.467+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd13c111ae0 con 0x7fd12406ec00 2024-04-16T07:27:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.469+0000 7fd138c00700 1 -- 172.21.15.26:0/578785810 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd13c111ae0 con 0x7fd12406ec00 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:27:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "7/21 daemons upgraded", 2024-04-16T07:27:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T07:27:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:27:05.469 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.471+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 msgr2=0x7fd1240710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.471+0000 7fd142200700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 0x7fd1240710b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd13000b7a0 tx=0x7fd130009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.471+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 msgr2=0x7fd13c110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.471+0000 7fd142200700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c830 tx=0x7fd128004470 comp rx=0 tx=0).stop 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 shutdown_connections 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd12406ec00 0x7fd1240710b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13c066c80 0x7fd13c10faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 --2- 172.21.15.26:0/578785810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd13c1021e0 0x7fd13c110030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 >> 172.21.15.26:0/578785810 conn(0x7fd13c0f72b0 msgr2=0x7fd13c0f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 shutdown_connections 2024-04-16T07:27:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.472+0000 7fd142200700 1 -- 172.21.15.26:0/578785810 wait complete. 2024-04-16T07:27:05.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 -- 172.21.15.26:0/4289767344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c0708f0 msgr2=0x7fcb6c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 --2- 172.21.15.26:0/4289767344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c0708f0 0x7fcb6c070d60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcb54004940 tx=0x7fcb54009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 -- 172.21.15.26:0/4289767344 shutdown_connections 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 --2- 172.21.15.26:0/4289767344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c0708f0 0x7fcb6c070d60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 --2- 172.21.15.26:0/4289767344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 -- 172.21.15.26:0/4289767344 >> 172.21.15.26:0/4289767344 conn(0x7fcb6c0f8580 msgr2=0x7fcb6c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 -- 172.21.15.26:0/4289767344 shutdown_connections 2024-04-16T07:27:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.622+0000 7fcb71c00700 1 -- 172.21.15.26:0/4289767344 wait complete. 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.623+0000 7fcb71c00700 1 Processor -- start 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.623+0000 7fcb71c00700 1 -- start start 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb71c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb71c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6c110290 0x7fcb6c115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb71c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c110750 con 0x7fcb6c071ea0 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb71c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1108c0 con 0x7fcb6c110290 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37780/0 (socket says 172.21.15.26:37780) 2024-04-16T07:27:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 -- 172.21.15.26:0/2276352772 learned_addr learned my addr 172.21.15.26:0/2276352772 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 -- 172.21.15.26:0/2276352772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6c110290 msgr2=0x7fcb6c115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6c110290 0x7fcb6c115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.624+0000 7fcb6aa00700 1 -- 172.21.15.26:0/2276352772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54005040 con 0x7fcb6c071ea0 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb6aa00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c8b0 tx=0x7fcb5c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c00a760 con 0x7fcb6c071ea0 2024-04-16T07:27:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb5c00ad90 con 0x7fcb6c071ea0 2024-04-16T07:27:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c0113a0 con 0x7fcb6c071ea0 2024-04-16T07:27:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c115820 con 0x7fcb6c071ea0 2024-04-16T07:27:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.625+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c115db0 con 0x7fcb6c071ea0 2024-04-16T07:27:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.627+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c04c710 con 0x7fcb6c071ea0 2024-04-16T07:27:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.628+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcb5c012400 con 0x7fcb6c071ea0 2024-04-16T07:27:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.629+0000 7fcb62a00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 0x7fcb500710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.629+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcb5c08e760 con 0x7fcb6c071ea0 2024-04-16T07:27:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.629+0000 7fcb63e00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 0x7fcb500710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.630+0000 7fcb63e00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 0x7fcb500710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400b3b0 tx=0x7fcb54009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.633+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcb5c057650 con 0x7fcb6c071ea0 2024-04-16T07:27:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.973+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcb6c116080 con 0x7fcb6c071ea0 2024-04-16T07:27:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.974+0000 7fcb62a00700 1 -- 172.21.15.26:0/2276352772 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fcb5c05ad10 con 0x7fcb6c071ea0 2024-04-16T07:27:05.973 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:27:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 msgr2=0x7fcb500710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 0x7fcb500710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400b3b0 tx=0x7fcb54009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 msgr2=0x7fcb6c10fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c8b0 tx=0x7fcb5c004470 comp rx=0 tx=0).stop 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 shutdown_connections 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcb5006ec00 0x7fcb500710b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb6c071ea0 0x7fcb6c10fd50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 --2- 172.21.15.26:0/2276352772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6c110290 0x7fcb6c115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.977+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 >> 172.21.15.26:0/2276352772 conn(0x7fcb6c0f8580 msgr2=0x7fcb6c0f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.978+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 shutdown_connections 2024-04-16T07:27:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:05.978+0000 7fcb71c00700 1 -- 172.21.15.26:0/2276352772 wait complete. 2024-04-16T07:27:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:05 smithi026 ceph-mon[92215]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:05 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T07:27:06.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:05 smithi026 ceph-mon[92215]: Upgrade: osd.1 is safe to restart 2024-04-16T07:27:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:05 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1816421946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:27:06.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:06.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:06.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:06.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:06.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:06.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:06.071+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:05 smithi104 ceph-mon[81027]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:05 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T07:27:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:05 smithi104 ceph-mon[81027]: Upgrade: osd.1 is safe to restart 2024-04-16T07:27:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:05 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1816421946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:27:06.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:06.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:06.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:06.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:06.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:06.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:06.359+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: pgmap v76: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2276352772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:27:06.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.071+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.071+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.071+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.071+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.072+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:07.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:07.072+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: pgmap v76: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2276352772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:27:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T07:27:07.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:07.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:07.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:07.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:07.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:07.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:07.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:07.359+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:07.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:07 smithi026 ceph-mon[92215]: Upgrade: Updating osd.1 2024-04-16T07:27:07.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:07 smithi026 ceph-mon[92215]: Deploying daemon osd.1 on smithi026 2024-04-16T07:27:08.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:08.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:08.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:08.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:08.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:08.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:08.072+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:08.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:07 smithi104 ceph-mon[81027]: Upgrade: Updating osd.1 2024-04-16T07:27:08.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:07 smithi104 ceph-mon[81027]: Deploying daemon osd.1 on smithi026 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:08.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:08.360+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:09.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:08 smithi026 ceph-mon[92215]: pgmap v77: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:09.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:09.072+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:09.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:08 smithi104 ceph-mon[81027]: pgmap v77: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:09.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:09.360+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:10.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:10.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:10.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:10.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:10.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:10.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:10.072+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:10.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:10.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:10.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:10.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:10.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:10.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:10.360+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:10.717 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:10 smithi026 ceph-mon[92215]: pgmap v78: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:10.717 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:10 smithi104 ceph-mon[81027]: pgmap v78: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:10.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.072+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.072+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.072+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.072+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.073+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:11.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:11.073+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:11.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:11.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:11.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:11.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:11.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:11.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:11.360+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:12.011 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:11 smithi026 systemd[1]: Stopping Ceph osd.1 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:12.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:12.073+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 22) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:27:12.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:12.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.361+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:12.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.362+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 26 ==== client_session(renewcaps seq 22) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:12.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:12.362+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 22) v1 from mds.0 2024-04-16T07:27:12.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[54579]: 2024-04-16T07:27:12.017+0000 7f4912600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:27:12.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[54579]: 2024-04-16T07:27:12.017+0000 7f4912600700 -1 osd.1 55 *** Got signal Terminated *** 2024-04-16T07:27:12.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[54579]: 2024-04-16T07:27:12.017+0000 7f4912600700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:27:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:12 smithi026 ceph-mon[92215]: pgmap v79: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:12 smithi026 ceph-mon[92215]: osd.1 marked itself down and dead 2024-04-16T07:27:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:12 smithi104 ceph-mon[81027]: pgmap v79: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:12 smithi104 ceph-mon[81027]: osd.1 marked itself down and dead 2024-04-16T07:27:13.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:13.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:13.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:13.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:13.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:13.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:13.073+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:13.175 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:12 smithi026 podman[106401]: 2024-04-16 07:27:12.841830007 +0000 UTC m=+0.950870895 container stop 99152f5e97760cb80bf55beac53ceb8052b20b64904eccf4a3243a1aafdaaf68 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-16T07:27:13.175 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:12 smithi026 podman[106401]: 2024-04-16 07:27:12.868203391 +0000 UTC m=+0.977244259 container died 99152f5e97760cb80bf55beac53ceb8052b20b64904eccf4a3243a1aafdaaf68 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:13.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:13.361+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:13.508 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:13 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:27:13.509 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:13 smithi026 ceph-mon[92215]: osdmap e56: 6 total, 5 up, 6 in 2024-04-16T07:27:13.509 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:13 smithi026 podman[106401]: 2024-04-16 07:27:13.284800502 +0000 UTC m=+1.393841407 container cleanup 99152f5e97760cb80bf55beac53ceb8052b20b64904eccf4a3243a1aafdaaf68 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:27:13.509 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:13 smithi026 bash[106401]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1 2024-04-16T07:27:13.760 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:13 smithi026 podman[106420]: 2024-04-16 07:27:13.536137106 +0000 UTC m=+0.679148328 container remove 99152f5e97760cb80bf55beac53ceb8052b20b64904eccf4a3243a1aafdaaf68 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-16T07:27:13.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:13 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:27:13.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:13 smithi104 ceph-mon[81027]: osdmap e56: 6 total, 5 up, 6 in 2024-04-16T07:27:14.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:14.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:14.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:14.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:14.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:14.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:14.073+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:14.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:14.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:14.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:14.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:14.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:14.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:14.361+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:14.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:14 smithi026 ceph-mon[92215]: pgmap v81: 65 pgs: 12 stale+active+clean, 53 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:14.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:14 smithi026 ceph-mon[92215]: osdmap e57: 6 total, 5 up, 6 in 2024-04-16T07:27:14.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:14 smithi026 podman[106505]: 2024-04-16 07:27:14.259890219 +0000 UTC m=+0.039241718 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:14.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:14 smithi026 podman[106505]: 2024-04-16 07:27:14.386989476 +0000 UTC m=+0.166340938 container create 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-16T07:27:14.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:14 smithi026 podman[106505]: 2024-04-16 07:27:14.666604587 +0000 UTC m=+0.445956092 container init 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-16T07:27:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:14 smithi104 ceph-mon[81027]: pgmap v81: 65 pgs: 12 stale+active+clean, 53 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:14 smithi104 ceph-mon[81027]: osdmap e57: 6 total, 5 up, 6 in 2024-04-16T07:27:15.011 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:14 smithi026 podman[106505]: 2024-04-16 07:27:14.678866554 +0000 UTC m=+0.458218073 container start 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:27:15.011 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:14 smithi026 podman[106505]: 2024-04-16 07:27:14.773168503 +0000 UTC m=+0.552520031 container attach 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:15.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:15.073+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:15.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:15.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:15.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:15.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:15.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:15.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:15.361+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:15.456 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:15 smithi026 podman[106505]: 2024-04-16 07:27:15.141024501 +0000 UTC m=+0.920375970 container died 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T07:27:15.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:15 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:27:15.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:15 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:27:16.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:16.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:16.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:16.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:16.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:16.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:16.074+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:16.225 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:15 smithi026 podman[106531]: 2024-04-16 07:27:15.810540363 +0000 UTC m=+0.649410029 container remove 0e5973bb3abf54d7668c98b3da1c2ba0d14639ee24b3b74c5f49a1809af2f844 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-04-16T07:27:16.225 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:15 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-16T07:27:16.225 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:15 smithi026 systemd[1]: Stopped Ceph osd.1 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:27:16.225 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:15 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.1.service: Consumed 13.609s CPU time. 2024-04-16T07:27:16.225 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 systemd[1]: Starting Ceph osd.1 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:27:16.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.361+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:16.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:16.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.361+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:16.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.362+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:16.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.362+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:16.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:16.362+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:16.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:16 smithi026 ceph-mon[92215]: pgmap v83: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:16.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:16 smithi104 ceph-mon[81027]: pgmap v83: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:16.873 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 podman[106630]: 2024-04-16 07:27:16.501808655 +0000 UTC m=+0.037315121 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:16.873 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 podman[106630]: 2024-04-16 07:27:16.594570828 +0000 UTC m=+0.130077331 container create 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:27:16.873 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 podman[106630]: 2024-04-16 07:27:16.873743528 +0000 UTC m=+0.409250037 container init 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T07:27:17.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:17.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:17.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:17.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 116 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:17.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:17.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.074+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:17.261 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 podman[106630]: 2024-04-16 07:27:16.885695511 +0000 UTC m=+0.421201986 container start 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T07:27:17.261 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:16 smithi026 podman[106630]: 2024-04-16 07:27:16.963766536 +0000 UTC m=+0.499273089 container attach 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:17.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:17.362+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:17.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106755 2024-04-16T07:27:17.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 238, success, outsize: 120 2024-04-16T07:27:17.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.668+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:17.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.668+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:17.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.668+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:17.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.668+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:17.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:17.668+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:18.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:18.074+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T07:27:18.256 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 bash[106630]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate[106644]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T07:27:18.257 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:17 smithi026 podman[106630]: 2024-04-16 07:27:17.873295955 +0000 UTC m=+1.408802478 container died 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:18.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:18.362+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:18.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:18 smithi026 ceph-mon[92215]: pgmap v84: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:18 smithi104 ceph-mon[81027]: pgmap v84: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:18.985 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:18 smithi026 podman[106783]: 2024-04-16 07:27:18.625968043 +0000 UTC m=+0.730730072 container remove 3637cf1e0958492d94d0e4120a1046348c05df877144141ac76d55d24d47a68e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.074+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.074+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.074+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.075+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.075+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:19.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:19.075+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:19.261 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:18 smithi026 podman[106843]: 2024-04-16 07:27:18.891876255 +0000 UTC m=+0.035527009 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:19.261 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 podman[106843]: 2024-04-16 07:27:19.001814592 +0000 UTC m=+0.145465383 container create 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-16T07:27:19.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:19.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:19.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:19.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:19.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:19.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:19.362+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:19.761 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 podman[106843]: 2024-04-16 07:27:19.264855322 +0000 UTC m=+0.408506183 container init 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:27:19.761 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 podman[106843]: 2024-04-16 07:27:19.278372838 +0000 UTC m=+0.422023614 container start 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:27:19.761 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 ceph-osd[106862]: -- 172.21.15.26:0/1448397979 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5641ca831680 con 0x5641ca7fb000 2024-04-16T07:27:19.761 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 bash[106843]: 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b 2024-04-16T07:27:19.761 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 systemd[1]: Started Ceph osd.1 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:27:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:20.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:20.075+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:20.261 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:19 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:27:19.858+0000 7f28bd3d97c0 -1 Falling back to public interface 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:20.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:20.363+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: pgmap v85: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: pgmap v85: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:20.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:21.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:21.075+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:21.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:21.363+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:22.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:22.076+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:22.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:22.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:22.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:22.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:22.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:22.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:22.363+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:22.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:22 smithi026 ceph-mon[92215]: pgmap v86: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:22 smithi104 ceph-mon[81027]: pgmap v86: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.076+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.076+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.076+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.077+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.077+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:23.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:23.077+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:23.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:23.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:23.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:23.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:23.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:23.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:23.363+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:23.760 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:23 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:27:23.507+0000 7f28bd3d97c0 -1 osd.1 55 log_to_monitors true 2024-04-16T07:27:24.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:24.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:27:24.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:27:24.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:27:24.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 23) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.077+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.078+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 28 ==== client_session(renewcaps seq 23) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:27:24.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:24.078+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 23) v1 from mds.0 2024-04-16T07:27:24.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:24.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:24.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:24.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:24.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:24.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:24.364+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:24.659 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:24 smithi026 ceph-mon[92215]: pgmap v87: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:24.659 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:24 smithi026 ceph-mon[92215]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T07:27:24.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:24 smithi104 ceph-mon[81027]: pgmap v87: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:24.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:24 smithi104 ceph-mon[81027]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T07:27:25.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:25.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:25.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:25.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:25.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:25.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:25.077+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:25.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:25.364+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:25.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:27:25 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:27:25.322+0000 7f28a7800700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:27:25.640 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:25 smithi026 ceph-mon[92215]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T07:27:25.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:25 smithi026 ceph-mon[92215]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T07:27:25.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:25 smithi026 ceph-mon[92215]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:27:25.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:25 smithi026 ceph-mon[92215]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' 2024-04-16T07:27:25.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:25 smithi104 ceph-mon[81027]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T07:27:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:25 smithi104 ceph-mon[81027]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T07:27:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:25 smithi104 ceph-mon[81027]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:27:25.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:25 smithi104 ceph-mon[81027]: from='osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504]' entity='osd.1' 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:26.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:26.077+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:26.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:26.364+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:26 smithi026 ceph-mon[92215]: pgmap v89: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:26 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:27:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:26 smithi026 ceph-mon[92215]: osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504] boot 2024-04-16T07:27:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:26 smithi026 ceph-mon[92215]: osdmap e59: 6 total, 6 up, 6 in 2024-04-16T07:27:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:26 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:27:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:26 smithi104 ceph-mon[81027]: pgmap v89: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:26 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:27:26.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:26 smithi104 ceph-mon[81027]: osd.1 [v2:172.21.15.26:6810/547286504,v1:172.21.15.26:6811/547286504] boot 2024-04-16T07:27:26.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:26 smithi104 ceph-mon[81027]: osdmap e59: 6 total, 6 up, 6 in 2024-04-16T07:27:26.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:26 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.077+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.077+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.077+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.078+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.078+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:27.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:27.078+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:27.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:27.364+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:27 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:27.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:27 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:27.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:27 smithi026 ceph-mon[92215]: osdmap e60: 6 total, 6 up, 6 in 2024-04-16T07:27:27.781 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:27 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:27.781 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:27 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:27.781 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:27 smithi104 ceph-mon[81027]: osdmap e60: 6 total, 6 up, 6 in 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:28.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:28.078+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:28.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:28.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:28.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:28.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:28.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:28.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:28.364+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:28.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:28 smithi026 ceph-mon[92215]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:28 smithi104 ceph-mon[81027]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:29.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:29.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:29.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:29.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:29.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:29.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:29.078+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:29.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:29.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:29.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:29.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:29.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:29.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:29.365+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:29 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:29.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:29 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:29.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:29 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:29.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:29 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:30.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:30.078+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:30.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:30.365+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:30.706 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:30 smithi026 ceph-mon[92215]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:30 smithi104 ceph-mon[81027]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:31.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:31.079+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:31.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:31.365+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:31.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:31 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-04-16T07:27:31.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:31 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:27:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:31 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-04-16T07:27:31.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:31 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 117 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:32.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.079+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:32.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:32.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:27:32.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 23) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.365+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.366+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 27 ==== client_session(renewcaps seq 23) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:32.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:32.366+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 23) v1 from mds.0 2024-04-16T07:27:32.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108792 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 240, success, outsize: 120 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.407+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.407+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.407+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.407+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.407+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108874 2024-04-16T07:27:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 242, success, outsize: 120 2024-04-16T07:27:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.441+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.441+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.441+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.441+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.441+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108959 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 244, success, outsize: 120 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.474+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.474+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.474+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.474+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.474+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109050 2024-04-16T07:27:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 246, success, outsize: 120 2024-04-16T07:27:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.508+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.508+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.508+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.508+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.508+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109138 2024-04-16T07:27:32.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 248, success, outsize: 120 2024-04-16T07:27:32.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.537+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.537+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.537+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.537+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.538+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109220 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 250, success, outsize: 120 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.568+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.568+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.568+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.568+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.568+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109305 2024-04-16T07:27:32.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 252, success, outsize: 120 2024-04-16T07:27:32.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.595+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.595+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.595+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.595+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.595+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109396 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 254, success, outsize: 120 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.627+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.627+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.627+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.627+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.627+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109484 2024-04-16T07:27:32.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.658+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 256, success, outsize: 120 2024-04-16T07:27:32.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.658+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.658+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.658+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.658+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109566 2024-04-16T07:27:32.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 258, success, outsize: 120 2024-04-16T07:27:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.687+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.687+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.687+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.687+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.687+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109651 2024-04-16T07:27:32.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 260, success, outsize: 120 2024-04-16T07:27:32.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.717+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.717+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.717+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.717+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.717+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109742 2024-04-16T07:27:32.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.751+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 262, success, outsize: 120 2024-04-16T07:27:32.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.751+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.751+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.751+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.751+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:32 smithi026 ceph-mon[92215]: pgmap v94: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:32.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109830 2024-04-16T07:27:32.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 264, success, outsize: 120 2024-04-16T07:27:32.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.785+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.785+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.785+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.785+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.785+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109912 2024-04-16T07:27:32.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 266, success, outsize: 120 2024-04-16T07:27:32.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.816+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.816+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.816+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.816+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.816+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:32 smithi104 ceph-mon[81027]: pgmap v94: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:32.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109997 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 268, success, outsize: 120 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.848+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.848+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.848+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.848+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.848+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110088 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 270, success, outsize: 120 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.877+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.877+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.877+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.877+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.877+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110176 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.909+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 272, success, outsize: 120 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.909+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.909+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.909+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.909+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110258 2024-04-16T07:27:32.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 274, success, outsize: 120 2024-04-16T07:27:32.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.937+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.937+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.937+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.937+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.937+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110343 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 276, success, outsize: 120 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.960+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.960+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.960+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.960+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.960+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110434 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 278, success, outsize: 120 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.986+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.986+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.986+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.986+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:32.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:32.986+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110522 2024-04-16T07:27:33.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 280, success, outsize: 120 2024-04-16T07:27:33.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.015+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.015+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.015+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.015+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.015+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110604 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 282, success, outsize: 120 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.041+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.041+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.041+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.041+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.041+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110689 2024-04-16T07:27:33.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 284, success, outsize: 120 2024-04-16T07:27:33.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.066+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.066+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.066+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.066+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.066+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 140 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.079+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110780 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 286, success, outsize: 120 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.096+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.096+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.096+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.096+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.096+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110868 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 288, success, outsize: 120 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.123+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.123+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.123+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.123+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.123+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110950 2024-04-16T07:27:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 290, success, outsize: 120 2024-04-16T07:27:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.150+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.150+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.150+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.150+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.150+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111035 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 292, success, outsize: 120 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.176+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.176+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.176+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.176+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.176+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111126 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 294, success, outsize: 120 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.199+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.199+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.199+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.199+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:33.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:33.199+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:33.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:33.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:33.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:33.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:33.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:33.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:33.366+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:34.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:34.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:34.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:34.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:34.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:34.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:34.079+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:34.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:34.366+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: pgmap v95: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:34.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:34.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:34.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:34.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:34.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: pgmap v95: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:35.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T07:27:35.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:35.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:35.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:35.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:35.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:35.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:35.079+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:35.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:35.366+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:35.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:35 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T07:27:35.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:35 smithi026 ceph-mon[92215]: Upgrade: osd.2 is safe to restart 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:36.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:36.080+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:35 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T07:27:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:35 smithi104 ceph-mon[81027]: Upgrade: osd.2 is safe to restart 2024-04-16T07:27:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 -- 172.21.15.26:0/2918175635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c0708f0 msgr2=0x7ffb9c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 --2- 172.21.15.26:0/2918175635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c070d60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb88004940 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 -- 172.21.15.26:0/2918175635 shutdown_connections 2024-04-16T07:27:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 --2- 172.21.15.26:0/2918175635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c070d60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 --2- 172.21.15.26:0/2918175635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.140+0000 7ffba3000700 1 -- 172.21.15.26:0/2918175635 >> 172.21.15.26:0/2918175635 conn(0x7ffb9c0f8530 msgr2=0x7ffb9c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.141+0000 7ffba3000700 1 -- 172.21.15.26:0/2918175635 shutdown_connections 2024-04-16T07:27:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.141+0000 7ffba3000700 1 -- 172.21.15.26:0/2918175635 wait complete. 2024-04-16T07:27:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.142+0000 7ffba3000700 1 Processor -- start 2024-04-16T07:27:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.142+0000 7ffba3000700 1 -- start start 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffba3000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffba3000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffba3000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c114b70 con 0x7ffb9c071ea0 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffba3000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c1153c0 con 0x7ffb9c0708f0 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffb9b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffb9b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41930/0 (socket says 172.21.15.26:41930) 2024-04-16T07:27:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffb9b400700 1 -- 172.21.15.26:0/847525990 learned_addr learned my addr 172.21.15.26:0/847525990 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:36.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.143+0000 7ffb9be00700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb9b400700 1 -- 172.21.15.26:0/847525990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c0708f0 msgr2=0x7ffb9c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb9b400700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb9b400700 1 -- 172.21.15.26:0/847525990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb88005040 con 0x7ffb9c071ea0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb9b400700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800b3b0 tx=0x7ffb88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8800e040 con 0x7ffb9c071ea0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c115640 con 0x7ffb9c071ea0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb88004290 con 0x7ffb9c071ea0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.144+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8801c630 con 0x7ffb9c071ea0 2024-04-16T07:27:36.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.145+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c115b80 con 0x7ffb9c071ea0 2024-04-16T07:27:36.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.146+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c101150 con 0x7ffb9c071ea0 2024-04-16T07:27:36.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.147+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffb8800bbf0 con 0x7ffb9c071ea0 2024-04-16T07:27:36.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.148+0000 7ffb93e00700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 0x7ffb800710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.148+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb8808fd20 con 0x7ffb9c071ea0 2024-04-16T07:27:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.149+0000 7ffb9be00700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 0x7ffb800710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.149+0000 7ffb9be00700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 0x7ffb800710c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009fd0 tx=0x7ffb8c009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:36.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.154+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffb8805c600 con 0x7ffb9c071ea0 2024-04-16T07:27:36.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:36.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:36.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:36.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:36.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:36.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:36.367+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:36.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.414+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb9c115e60 con 0x7ffb8006ec10 2024-04-16T07:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.422+0000 7ffb93e00700 1 -- 172.21.15.26:0/847525990 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ffb9c115e60 con 0x7ffb8006ec10 2024-04-16T07:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 msgr2=0x7ffb800710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 0x7ffb800710c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009fd0 tx=0x7ffb8c009f00 comp rx=0 tx=0).stop 2024-04-16T07:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 msgr2=0x7ffb9c114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800b3b0 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 shutdown_connections 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ffb8006ec10 0x7ffb800710c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb9c0708f0 0x7ffb9c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 --2- 172.21.15.26:0/847525990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb9c071ea0 0x7ffb9c114570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 >> 172.21.15.26:0/847525990 conn(0x7ffb9c0f8530 msgr2=0x7ffb9c0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 shutdown_connections 2024-04-16T07:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.425+0000 7ffba3000700 1 -- 172.21.15.26:0/847525990 wait complete. 2024-04-16T07:27:36.440 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 -- 172.21.15.26:0/1651535289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 msgr2=0x7f1d38104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 --2- 172.21.15.26:0/1651535289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38104b40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d24004940 tx=0x7f1d24009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 -- 172.21.15.26:0/1651535289 shutdown_connections 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 --2- 172.21.15.26:0/1651535289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38104b40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 --2- 172.21.15.26:0/1651535289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 -- 172.21.15.26:0/1651535289 >> 172.21.15.26:0/1651535289 conn(0x7f1d38072640 msgr2=0x7f1d38074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:36.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 -- 172.21.15.26:0/1651535289 shutdown_connections 2024-04-16T07:27:36.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.584+0000 7f1d40600700 1 -- 172.21.15.26:0/1651535289 wait complete. 2024-04-16T07:27:36.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.585+0000 7f1d40600700 1 Processor -- start 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.585+0000 7f1d40600700 1 -- start start 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.585+0000 7f1d40600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.585+0000 7f1d40600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d40600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38114a90 con 0x7f1d381046d0 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d40600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38114c00 con 0x7f1d38103420 2024-04-16T07:27:36.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41936/0 (socket says 172.21.15.26:41936) 2024-04-16T07:27:36.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 -- 172.21.15.26:0/1299784006 learned_addr learned my addr 172.21.15.26:0/1299784006 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:36.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3dc00700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 -- 172.21.15.26:0/1299784006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 msgr2=0x7f1d38113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3d200700 1 -- 172.21.15.26:0/1299784006 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24005040 con 0x7f1d381046d0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.586+0000 7f1d3dc00700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d3d200700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009150 tx=0x7f1d24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2400e040 con 0x7f1d381046d0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d2400bcc0 con 0x7f1d381046d0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2401c630 con 0x7f1d381046d0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d38119600 con 0x7f1d381046d0 2024-04-16T07:27:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.587+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d38119b70 con 0x7f1d381046d0 2024-04-16T07:27:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.589+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3804c710 con 0x7f1d381046d0 2024-04-16T07:27:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.590+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1d24013420 con 0x7f1d381046d0 2024-04-16T07:27:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.591+0000 7f1d35600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 0x7f1d200710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.591+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1d2408f500 con 0x7f1d381046d0 2024-04-16T07:27:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.592+0000 7f1d3dc00700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 0x7f1d200710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:36.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.592+0000 7f1d3dc00700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 0x7f1d200710b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009fd0 tx=0x7f1d2c00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.596+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1d2405be60 con 0x7f1d381046d0 2024-04-16T07:27:36.720 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:36 smithi026 ceph-mon[92215]: pgmap v96: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.854+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d38108a10 con 0x7f1d2006ec00 2024-04-16T07:27:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.856+0000 7f1d35600700 1 -- 172.21.15.26:0/1299784006 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1d38108a10 con 0x7f1d2006ec00 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 msgr2=0x7f1d200710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 0x7f1d200710b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009fd0 tx=0x7f1d2c00b040 comp rx=0 tx=0).stop 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 msgr2=0x7f1d38114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009150 tx=0x7f1d24009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 shutdown_connections 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f1d2006ec00 0x7f1d200710b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d38103420 0x7f1d38113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 --2- 172.21.15.26:0/1299784006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d381046d0 0x7f1d38114490 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.858+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 >> 172.21.15.26:0/1299784006 conn(0x7f1d38072640 msgr2=0x7f1d38073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.859+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 shutdown_connections 2024-04-16T07:27:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:36.859+0000 7f1d40600700 1 -- 172.21.15.26:0/1299784006 wait complete. 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 -- 172.21.15.26:0/2051346280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 msgr2=0x7efc840fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 --2- 172.21.15.26:0/2051346280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc840fd040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc78004940 tx=0x7efc78009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 -- 172.21.15.26:0/2051346280 shutdown_connections 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 --2- 172.21.15.26:0/2051346280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc840fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 --2- 172.21.15.26:0/2051346280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc840fd040 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 -- 172.21.15.26:0/2051346280 >> 172.21.15.26:0/2051346280 conn(0x7efc840f83d0 msgr2=0x7efc840fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.034+0000 7efc8a600700 1 -- 172.21.15.26:0/2051346280 shutdown_connections 2024-04-16T07:27:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.035+0000 7efc8a600700 1 -- 172.21.15.26:0/2051346280 wait complete. 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.035+0000 7efc8a600700 1 Processor -- start 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc8a600700 1 -- start start 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc8a600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc8a600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc84070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc8a600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc841114b0 con 0x7efc840fcc30 2024-04-16T07:27:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc8a600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84111620 con 0x7efc840fdee0 2024-04-16T07:27:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc89200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc89200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41958/0 (socket says 172.21.15.26:41958) 2024-04-16T07:27:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.036+0000 7efc89200700 1 -- 172.21.15.26:0/2694036147 learned_addr learned my addr 172.21.15.26:0/2694036147 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc83e00700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc84070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc89200700 1 -- 172.21.15.26:0/2694036147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 msgr2=0x7efc84070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc89200700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc84070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc89200700 1 -- 172.21.15.26:0/2694036147 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78005040 con 0x7efc840fcc30 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc83e00700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc84070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.037+0000 7efc89200700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc78009150 tx=0x7efc78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.038+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7800e040 con 0x7efc840fcc30 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.038+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc7800bcc0 con 0x7efc840fcc30 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.038+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84070910 con 0x7efc840fcc30 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.038+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7801c630 con 0x7efc840fcc30 2024-04-16T07:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.038+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc84070e80 con 0x7efc840fcc30 2024-04-16T07:27:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.039+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8405f5f0 con 0x7efc840fcc30 2024-04-16T07:27:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.041+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efc78013420 con 0x7efc840fcc30 2024-04-16T07:27:37.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.042+0000 7efc80c00700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 0x7efc6c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.042+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc7808f4f0 con 0x7efc840fcc30 2024-04-16T07:27:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.043+0000 7efc83e00700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 0x7efc6c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.043+0000 7efc83e00700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 0x7efc6c0710b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc700095c0 tx=0x7efc70007f20 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.047+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efc78058320 con 0x7efc840fcc30 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:37.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:37.080+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:36 smithi104 ceph-mon[81027]: pgmap v96: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:37.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.284+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc84102200 con 0x7efc6c06ec00 2024-04-16T07:27:37.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.298+0000 7efc80c00700 1 -- 172.21.15.26:0/2694036147 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efc84102200 con 0x7efc6c06ec00 2024-04-16T07:27:37.298 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (4m) 10s ago 14m 13.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (86s) 10s ago 15m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (79s) 74s ago 13m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (4m) 10s ago 14m 72.1M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (7m) 10s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (7m) 10s ago 7m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (7m) 74s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (7m) 74s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (4m) 10s ago 16m 500M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (3m) 74s ago 13m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (2m) 10s ago 16m 44.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (2m) 74s ago 13m 30.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (4m) 10s ago 15m 7708k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (4m) 74s ago 13m 7578k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (48s) 10s ago 12m 48.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:27:37.299 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (18s) 10s ago 11m 11.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:27:37.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (11m) 10s ago 11m 47.4M 1559M 17.2.7-847-g45317f95 b333581ccdf5 c030235da07d 2024-04-16T07:27:37.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (10m) 74s ago 10m 46.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:27:37.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (10m) 74s ago 10m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:27:37.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (9m) 74s ago 9m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:27:37.300 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (3m) 10s ago 14m 57.2M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 msgr2=0x7efc6c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 0x7efc6c0710b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc700095c0 tx=0x7efc70007f20 comp rx=0 tx=0).stop 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 msgr2=0x7efc84071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc78009150 tx=0x7efc78009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 shutdown_connections 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efc6c06ec00 0x7efc6c0710b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc840fcc30 0x7efc84071e40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 --2- 172.21.15.26:0/2694036147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc840fdee0 0x7efc84070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 >> 172.21.15.26:0/2694036147 conn(0x7efc840f83d0 msgr2=0x7efc840f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:37.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.301+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 shutdown_connections 2024-04-16T07:27:37.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.302+0000 7efc8a600700 1 -- 172.21.15.26:0/2694036147 wait complete. 2024-04-16T07:27:37.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.366+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:37.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:37.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.366+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:37.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.366+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:37.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.367+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:37.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:37.367+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:37.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.454+0000 7ff7f4600700 1 -- 172.21.15.26:0/1972112827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec071e20 msgr2=0x7ff7ec070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.454+0000 7ff7f4600700 1 --2- 172.21.15.26:0/1972112827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec071e20 0x7ff7ec070330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0004940 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 -- 172.21.15.26:0/1972112827 shutdown_connections 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 --2- 172.21.15.26:0/1972112827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec070870 0x7ff7ec070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 --2- 172.21.15.26:0/1972112827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec071e20 0x7ff7ec070330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 -- 172.21.15.26:0/1972112827 >> 172.21.15.26:0/1972112827 conn(0x7ff7ec0f8530 msgr2=0x7ff7ec0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 -- 172.21.15.26:0/1972112827 shutdown_connections 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.455+0000 7ff7f4600700 1 -- 172.21.15.26:0/1972112827 wait complete. 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 Processor -- start 2024-04-16T07:27:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 -- start start 2024-04-16T07:27:37.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec071e20 0x7ff7ec1145b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec114bb0 con 0x7ff7ec070870 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.456+0000 7ff7f4600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec115400 con 0x7ff7ec071e20 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41974/0 (socket says 172.21.15.26:41974) 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 -- 172.21.15.26:0/3257004462 learned_addr learned my addr 172.21.15.26:0/3257004462 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1200700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec071e20 0x7ff7ec1145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 -- 172.21.15.26:0/3257004462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec071e20 msgr2=0x7ff7ec1145b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec071e20 0x7ff7ec1145b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 -- 172.21.15.26:0/3257004462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0005040 con 0x7ff7ec070870 2024-04-16T07:27:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.457+0000 7ff7f1c00700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000b970 tx=0x7ff7e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.458+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e000e040 con 0x7ff7ec070870 2024-04-16T07:27:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.458+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7e000bbf0 con 0x7ff7ec070870 2024-04-16T07:27:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.458+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e001c630 con 0x7ff7ec070870 2024-04-16T07:27:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.458+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec115680 con 0x7ff7ec070870 2024-04-16T07:27:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.458+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec115ab0 con 0x7ff7ec070870 2024-04-16T07:27:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.459+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec1011d0 con 0x7ff7ec070870 2024-04-16T07:27:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.460+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff7e000bd60 con 0x7ff7ec070870 2024-04-16T07:27:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.462+0000 7ff7e9600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 0x7ff7d40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.462+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff7e008fcb0 con 0x7ff7ec070870 2024-04-16T07:27:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.462+0000 7ff7f1200700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 0x7ff7d40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.463+0000 7ff7f1200700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 0x7ff7d40710c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8007fd0 tx=0x7ff7d8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.467+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff7e005c590 con 0x7ff7ec070870 2024-04-16T07:27:37.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.804+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff7ec1011d0 con 0x7ff7ec070870 2024-04-16T07:27:37.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.805+0000 7ff7e9600700 1 -- 172.21.15.26:0/3257004462 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7ff7e005c120 con 0x7ff7ec070870 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:27:37.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 8, 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:27:37.806 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 msgr2=0x7ff7d40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 0x7ff7d40710c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8007fd0 tx=0x7ff7d8009040 comp rx=0 tx=0).stop 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 msgr2=0x7ff7ec114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000b970 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 shutdown_connections 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff7d406ec10 0x7ff7d40710c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7ec070870 0x7ff7ec114070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 --2- 172.21.15.26:0/3257004462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7ec071e20 0x7ff7ec1145b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.808+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 >> 172.21.15.26:0/3257004462 conn(0x7ff7ec0f8530 msgr2=0x7ff7ec0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.809+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 shutdown_connections 2024-04-16T07:27:37.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.809+0000 7ff7f4600700 1 -- 172.21.15.26:0/3257004462 wait complete. 2024-04-16T07:27:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.966+0000 7fb3ef000700 1 -- 172.21.15.26:0/2085362704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fcc30 msgr2=0x7fb3e80fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.966+0000 7fb3ef000700 1 --2- 172.21.15.26:0/2085362704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fcc30 0x7fb3e80fd040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc004940 tx=0x7fb3dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 -- 172.21.15.26:0/2085362704 shutdown_connections 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 --2- 172.21.15.26:0/2085362704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fdee0 0x7fb3e80fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 --2- 172.21.15.26:0/2085362704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fcc30 0x7fb3e80fd040 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 -- 172.21.15.26:0/2085362704 >> 172.21.15.26:0/2085362704 conn(0x7fb3e80f83d0 msgr2=0x7fb3e80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 -- 172.21.15.26:0/2085362704 shutdown_connections 2024-04-16T07:27:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.967+0000 7fb3ef000700 1 -- 172.21.15.26:0/2085362704 wait complete. 2024-04-16T07:27:37.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:37.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: Upgrade: Updating osd.2 2024-04-16T07:27:37.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:37.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:27:37.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:37.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: Deploying daemon osd.2 on smithi026 2024-04-16T07:27:37.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:37 smithi026 ceph-mon[92215]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:37.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 Processor -- start 2024-04-16T07:27:37.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 -- start start 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fcc30 0x7fb3e810fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e81105d0 con 0x7fb3e80fdee0 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.968+0000 7fb3ef000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8110e20 con 0x7fb3e80fcc30 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42000/0 (socket says 172.21.15.26:42000) 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 -- 172.21.15.26:0/3470006617 learned_addr learned my addr 172.21.15.26:0/3470006617 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3edc00700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fcc30 0x7fb3e810fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 -- 172.21.15.26:0/3470006617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fcc30 msgr2=0x7fb3e810fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fcc30 0x7fb3e810fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 -- 172.21.15.26:0/3470006617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc005040 con 0x7fb3e80fdee0 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.969+0000 7fb3ed200700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400c830 tx=0x7fb3d4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.970+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d400a760 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.970+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d400ad90 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.970+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e8111100 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.970+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d40113a0 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.970+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8111640 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.971+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e8101870 con 0x7fb3e80fdee0 2024-04-16T07:27:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.972+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb3d4012400 con 0x7fb3e80fdee0 2024-04-16T07:27:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.973+0000 7fb3e5600700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 0x7fb3d00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.973+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3d408e890 con 0x7fb3e80fdee0 2024-04-16T07:27:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.974+0000 7fb3edc00700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 0x7fb3d00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:37.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.975+0000 7fb3edc00700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 0x7fb3d00710c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b3b0 tx=0x7fb3dc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:37.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:37.978+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb3d405b1f0 con 0x7fb3e80fdee0 2024-04-16T07:27:38.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:38.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:38.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:38.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:38.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:38.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:38.080+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:38.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:38.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: Upgrade: Updating osd.2 2024-04-16T07:27:38.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:38.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T07:27:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:27:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: Deploying daemon osd.2 on smithi026 2024-04-16T07:27:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:37 smithi104 ceph-mon[81027]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.283+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb3e805f5f0 con 0x7fb3e80fdee0 2024-04-16T07:27:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.283+0000 7fb3e5600700 1 -- 172.21.15.26:0/3470006617 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fb3d405ad80 con 0x7fb3e80fdee0 2024-04-16T07:27:38.283 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:27:38.283 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:27:38.284 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:27:38.285 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:38.286 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 msgr2=0x7fb3d00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 0x7fb3d00710c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b3b0 tx=0x7fb3dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 msgr2=0x7fb3e810ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400c830 tx=0x7fb3d4004470 comp rx=0 tx=0).stop 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 shutdown_connections 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb3d006ec10 0x7fb3d00710c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3e80fcc30 0x7fb3e810fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 --2- 172.21.15.26:0/3470006617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3e80fdee0 0x7fb3e810ffd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 >> 172.21.15.26:0/3470006617 conn(0x7fb3e80f83d0 msgr2=0x7fb3e80f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 shutdown_connections 2024-04-16T07:27:38.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.286+0000 7fb3ef000700 1 -- 172.21.15.26:0/3470006617 wait complete. 2024-04-16T07:27:38.288 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:38.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:38.367+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2549861115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34070870 msgr2=0x7f6c34070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2549861115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34070870 0x7f6c34070ce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c20004940 tx=0x7f6c20009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2549861115 shutdown_connections 2024-04-16T07:27:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2549861115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34070870 0x7f6c34070ce0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2549861115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34071e20 0x7f6c34070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2549861115 >> 172.21.15.26:0/2549861115 conn(0x7f6c340f8530 msgr2=0x7f6c340fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2549861115 shutdown_connections 2024-04-16T07:27:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.432+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2549861115 wait complete. 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 Processor -- start 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 -- start start 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34070870 0x7f6c3410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34110770 con 0x7f6c34071e20 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.433+0000 7f6c3ac00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34110fc0 con 0x7f6c34070870 2024-04-16T07:27:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47582/0 (socket says 172.21.15.26:47582) 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 -- 172.21.15.26:0/2807446387 learned_addr learned my addr 172.21.15.26:0/2807446387 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33e00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34070870 0x7f6c3410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 -- 172.21.15.26:0/2807446387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34070870 msgr2=0x7f6c3410fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34070870 0x7f6c3410fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.434+0000 7f6c33400700 1 -- 172.21.15.26:0/2807446387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c20005040 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c33400700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b3b0 tx=0x7f6c20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2000e040 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c2000bcc0 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2001c630 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c34111240 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.435+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c34111780 con 0x7f6c34071e20 2024-04-16T07:27:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.436+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3405f5f0 con 0x7f6c34071e20 2024-04-16T07:27:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.437+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c20013980 con 0x7f6c34071e20 2024-04-16T07:27:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.438+0000 7f6c2be00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 0x7f6c180710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.438+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6c2008f570 con 0x7f6c34071e20 2024-04-16T07:27:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.438+0000 7f6c33e00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 0x7f6c180710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.439+0000 7f6c33e00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 0x7f6c180710b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007fd0 tx=0x7f6c24009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:38.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.447+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6c2005be50 con 0x7f6c34071e20 2024-04-16T07:27:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.698+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c34111b70 con 0x7f6c1806ec00 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.700+0000 7f6c2be00700 1 -- 172.21.15.26:0/2807446387 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f6c34111b70 con 0x7f6c1806ec00 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:27:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "8/21 daemons upgraded", 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:27:38.700 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.702+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 msgr2=0x7f6c180710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.702+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 0x7f6c180710b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007fd0 tx=0x7f6c24009040 comp rx=0 tx=0).stop 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.702+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 msgr2=0x7f6c34110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.702+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b3b0 tx=0x7f6c20009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 shutdown_connections 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c1806ec00 0x7f6c180710b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c34070870 0x7f6c3410fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 --2- 172.21.15.26:0/2807446387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c34071e20 0x7f6c34110170 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 >> 172.21.15.26:0/2807446387 conn(0x7f6c340f8530 msgr2=0x7f6c340f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:38.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 shutdown_connections 2024-04-16T07:27:38.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.703+0000 7f6c3ac00700 1 -- 172.21.15.26:0/2807446387 wait complete. 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 -- 172.21.15.26:0/1139452366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fe040 msgr2=0x7fbcb40fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 --2- 172.21.15.26:0/1139452366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fe040 0x7fbcb40fe4b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004940 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 -- 172.21.15.26:0/1139452366 shutdown_connections 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 --2- 172.21.15.26:0/1139452366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fe040 0x7fbcb40fe4b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 --2- 172.21.15.26:0/1139452366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb40fce40 0x7fbcb40fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 -- 172.21.15.26:0/1139452366 >> 172.21.15.26:0/1139452366 conn(0x7fbcb40f8530 msgr2=0x7fbcb40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 -- 172.21.15.26:0/1139452366 shutdown_connections 2024-04-16T07:27:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.846+0000 7fbcba000700 1 -- 172.21.15.26:0/1139452366 wait complete. 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.847+0000 7fbcba000700 1 Processor -- start 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.847+0000 7fbcba000700 1 -- start start 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcba000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcba000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb40fe040 0x7fbcb4110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcba000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4110850 con 0x7fbcb40fce40 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcba000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4110f90 con 0x7fbcb40fe040 2024-04-16T07:27:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47600/0 (socket says 172.21.15.26:47600) 2024-04-16T07:27:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 -- 172.21.15.26:0/3443900181 learned_addr learned my addr 172.21.15.26:0/3443900181 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:27:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 -- 172.21.15.26:0/3443900181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb40fe040 msgr2=0x7fbcb4110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb40fe040 0x7fbcb4110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.848+0000 7fbcb3400700 1 -- 172.21.15.26:0/3443900181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca0005040 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcb3400700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbca400c8e0 tx=0x7fbca40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca400a940 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbca400ce80 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb4111270 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca40113a0 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.849+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb4111830 con 0x7fbcb40fce40 2024-04-16T07:27:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.850+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb41019d0 con 0x7fbcb40fce40 2024-04-16T07:27:38.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.851+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbca400aaa0 con 0x7fbcb40fce40 2024-04-16T07:27:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.852+0000 7fbcab400700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 0x7fbc980710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:27:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.852+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbca408e960 con 0x7fbcb40fce40 2024-04-16T07:27:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.853+0000 7fbcb2a00700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 0x7fbc980710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:27:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.853+0000 7fbcb2a00700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 0x7fbc980710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004f90 tx=0x7fbca0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:27:38.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:38.857+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbca405b240 con 0x7fbcb40fce40 2024-04-16T07:27:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:38 smithi026 ceph-mon[92215]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:38 smithi026 ceph-mon[92215]: pgmap v97: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:38 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3257004462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:38 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3470006617' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:27:39.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:39.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:39.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:39.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:39.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:39.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:39.080+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:38 smithi104 ceph-mon[81027]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:38 smithi104 ceph-mon[81027]: pgmap v97: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:38 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3257004462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:27:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:38 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3470006617' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:27:39.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.172+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbcb405f5f0 con 0x7fbcb40fce40 2024-04-16T07:27:39.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.173+0000 7fbcab400700 1 -- 172.21.15.26:0/3443900181 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fbca4021250 con 0x7fbcb40fce40 2024-04-16T07:27:39.172 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 msgr2=0x7fbc980710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 0x7fbc980710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004f90 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 msgr2=0x7fbcb410fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbca400c8e0 tx=0x7fbca40044c0 comp rx=0 tx=0).stop 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 shutdown_connections 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbc9806ec10 0x7fbc980710c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcb40fce40 0x7fbcb410fd10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 --2- 172.21.15.26:0/3443900181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb40fe040 0x7fbcb4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.175+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 >> 172.21.15.26:0/3443900181 conn(0x7fbcb40f8530 msgr2=0x7fbcb40f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.176+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 shutdown_connections 2024-04-16T07:27:39.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:27:39.176+0000 7fbcba000700 1 -- 172.21.15.26:0/3443900181 wait complete. 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:39.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:39.367+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:40.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:40.080+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:40.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:39 smithi104 ceph-mon[81027]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:40.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:39 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3443900181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:27:40.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:39 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:39 smithi026 ceph-mon[92215]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:27:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:39 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3443900181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:27:40.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:39 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:40.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:40.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:40.366 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:40.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:40.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:40.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:40.368+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:41.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:40 smithi026 ceph-mon[92215]: pgmap v98: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:41.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:41.081+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:41.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:40 smithi104 ceph-mon[81027]: pgmap v98: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:41.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:41.368+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:41.998 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:41 smithi026 systemd[1]: Stopping Ceph osd.2 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:27:42.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:42.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:42.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:42.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:42.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:42.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:42.081+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:42.261 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[61883]: 2024-04-16T07:27:41.997+0000 7fb942800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:27:42.261 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[61883]: 2024-04-16T07:27:41.997+0000 7fb942800700 -1 osd.2 60 *** Got signal Terminated *** 2024-04-16T07:27:42.261 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:41 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[61883]: 2024-04-16T07:27:41.997+0000 7fb942800700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:42.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:42.368+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:43.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:43.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:43.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:43.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:43.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:43.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:43.082+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:43.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:42 smithi026 ceph-mon[92215]: pgmap v99: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:43.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:42 smithi026 ceph-mon[92215]: osd.2 marked itself down and dead 2024-04-16T07:27:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:42 smithi104 ceph-mon[81027]: pgmap v99: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:42 smithi104 ceph-mon[81027]: osd.2 marked itself down and dead 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:43.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:43.368+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:43.580 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:43 smithi026 podman[111762]: 2024-04-16 07:27:43.29645837 +0000 UTC m=+1.431327304 container died c030235da07d052062cda207bfe22120bdbc7bf6c9d756d37080b4141386e22f (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-16T07:27:43.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:43 smithi026 podman[111762]: 2024-04-16 07:27:43.715337772 +0000 UTC m=+1.850206709 container cleanup c030235da07d052062cda207bfe22120bdbc7bf6c9d756d37080b4141386e22f (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-16T07:27:43.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:43 smithi026 bash[111762]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 24) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:44.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:44.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:44.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.081+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:44.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.082+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 29 ==== client_session(renewcaps seq 24) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:27:44.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:44.082+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 24) v1 from mds.0 2024-04-16T07:27:44.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:43 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:27:44.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:43 smithi026 ceph-mon[92215]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T07:27:44.261 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:43 smithi026 podman[111782]: 2024-04-16 07:27:43.916631086 +0000 UTC m=+0.601658442 container remove c030235da07d052062cda207bfe22120bdbc7bf6c9d756d37080b4141386e22f (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-16T07:27:44.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:43 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:27:44.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:43 smithi104 ceph-mon[81027]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T07:27:44.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:44.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:44.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:44.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:44.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:44.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:44.369+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:45.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:44 smithi026 ceph-mon[92215]: pgmap v101: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:45.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:44 smithi026 ceph-mon[92215]: osdmap e62: 6 total, 5 up, 6 in 2024-04-16T07:27:45.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:44 smithi026 podman[111870]: 2024-04-16 07:27:44.642844072 +0000 UTC m=+0.034795449 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:45.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:44 smithi026 podman[111870]: 2024-04-16 07:27:44.767591282 +0000 UTC m=+0.159542697 container create 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:45.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:45.082+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:45.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:44 smithi104 ceph-mon[81027]: pgmap v101: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T07:27:45.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:44 smithi104 ceph-mon[81027]: osdmap e62: 6 total, 5 up, 6 in 2024-04-16T07:27:45.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.368+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:45.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:45.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:45.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.369+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:45.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.369+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:45.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:45.369+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:45.502 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:45 smithi026 podman[111870]: 2024-04-16 07:27:45.022166015 +0000 UTC m=+0.414117437 container init 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-16T07:27:45.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:45 smithi026 podman[111870]: 2024-04-16 07:27:45.03528562 +0000 UTC m=+0.427237042 container start 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-16T07:27:45.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:45 smithi026 podman[111870]: 2024-04-16 07:27:45.132979578 +0000 UTC m=+0.524930999 container attach 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T07:27:45.760 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:45 smithi026 podman[111870]: 2024-04-16 07:27:45.504940741 +0000 UTC m=+0.896892163 container died 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-16T07:27:46.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:46.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:46.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:46.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:46.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:46.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:46.082+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:46.174 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:45 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T07:27:46.234 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:45 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:46.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:46.369+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:46.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:46 smithi026 podman[111896]: 2024-04-16 07:27:46.173904045 +0000 UTC m=+0.649643021 container remove 6adef5b3330142536d1e5ddc5fa386d8fe4b08b04643f7a646983534f5d3801c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:27:46.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:46 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-16T07:27:46.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:46 smithi026 systemd[1]: Stopped Ceph osd.2 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:27:46.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:46 smithi026 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.2.service: Consumed 13.144s CPU time. 2024-04-16T07:27:46.914 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:46 smithi026 systemd[1]: Starting Ceph osd.2 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:27:47.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:47.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:47.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:47.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 145 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:47.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:47.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:47.082+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:47.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:46 smithi026 ceph-mon[92215]: pgmap v103: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:47 smithi026 podman[111997]: 2024-04-16 07:27:47.008192039 +0000 UTC m=+0.123979487 container create 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:27:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:47 smithi026 podman[111997]: 2024-04-16 07:27:46.917682202 +0000 UTC m=+0.033469620 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:47.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:46 smithi104 ceph-mon[81027]: pgmap v103: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:47.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:47.370+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:47.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:47 smithi026 podman[111997]: 2024-04-16 07:27:47.271173872 +0000 UTC m=+0.386961367 container init 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-16T07:27:47.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:47 smithi026 podman[111997]: 2024-04-16 07:27:47.283473166 +0000 UTC m=+0.399260594 container start 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:27:47.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:47 smithi026 podman[111997]: 2024-04-16 07:27:47.377576911 +0000 UTC m=+0.493364406 container attach 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:27:48.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112091 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 296, success, outsize: 120 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.056+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.056+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.056+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.056+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:27:48.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.056+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:27:48.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:48.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:48.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:48.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:48.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:48.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:48.082+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:48.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:48.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:48.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:48.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:48.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:48.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:48.370+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:48.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 bash[111997]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate[112011]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T07:27:48.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 podman[111997]: 2024-04-16 07:27:48.220939007 +0000 UTC m=+1.336726462 container died 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.082+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.082+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.082+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.082+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.083+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:49.083+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:49.260 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:48 smithi026 podman[112135]: 2024-04-16 07:27:48.981516005 +0000 UTC m=+0.741815527 container remove 0faef3605502445f9d199cf7e35433ce444d95661038e0e5d6a9d1cf70939464 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T07:27:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:48 smithi026 ceph-mon[92215]: pgmap v104: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:48 smithi104 ceph-mon[81027]: pgmap v104: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:49.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:49.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:49.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:49.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:49.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:49.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:49.370+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:49.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 podman[112194]: 2024-04-16 07:27:49.242547908 +0000 UTC m=+0.035321730 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:27:49.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 podman[112194]: 2024-04-16 07:27:49.365815201 +0000 UTC m=+0.158589087 container create 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T07:27:49.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 podman[112194]: 2024-04-16 07:27:49.644626262 +0000 UTC m=+0.437400128 container init 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:27:50.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 podman[112194]: 2024-04-16 07:27:49.656892197 +0000 UTC m=+0.449666031 container start 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T07:27:50.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 ceph-osd[112212]: -- 172.21.15.26:0/2259231777 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef20853680 con 0x55ef2081cc00 2024-04-16T07:27:50.012 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 bash[112194]: 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 2024-04-16T07:27:50.012 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:49 smithi026 systemd[1]: Started Ceph osd.2 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:50.083+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:50.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:50.370+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:50.510 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:27:50.234+0000 7f7f19eb67c0 -1 Falling back to public interface 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: pgmap v105: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: pgmap v105: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:50.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:27:51.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:51.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:51.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:51.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:51.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:51.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:51.083+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:51.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:51.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:51.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:51.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:51.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:51.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:51.371+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:52.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:52.083+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:52.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 24) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:52.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.370+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:52.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.371+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 28 ==== client_session(renewcaps seq 24) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:52.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:52.371+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 24) v1 from mds.0 2024-04-16T07:27:52.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:52 smithi026 ceph-mon[92215]: pgmap v106: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:52.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:52 smithi104 ceph-mon[81027]: pgmap v106: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:53.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:53.083+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:53.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:53.371+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.083+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.083+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.083+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.083+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.084+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:54.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:54.084+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:54.260 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:53 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:27:53.871+0000 7f7f19eb67c0 -1 osd.2 60 log_to_monitors true 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:54.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:54.371+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:54 smithi026 ceph-mon[92215]: pgmap v107: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:54.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:54 smithi026 ceph-mon[92215]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T07:27:54.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:27:54 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:27:54.381+0000 7f7f09200700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:27:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:54 smithi104 ceph-mon[81027]: pgmap v107: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:54.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:54 smithi104 ceph-mon[81027]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T07:27:55.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:55.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:55.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:55.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:55.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:55.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:55.084+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:55.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:55.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:55.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:55.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:55.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:55.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:55.371+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:55.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:55 smithi026 ceph-mon[92215]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T07:27:55.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:55 smithi026 ceph-mon[92215]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T07:27:55.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:55 smithi026 ceph-mon[92215]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:27:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:55 smithi104 ceph-mon[81027]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T07:27:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:55 smithi104 ceph-mon[81027]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T07:27:55.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:55 smithi104 ceph-mon[81027]: from='osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-16T07:27:56.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:56.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:56.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:56.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:56.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:56.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:56.084+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:56.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:56.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:56.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:56.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:56.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:56.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:56.371+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:56 smithi026 ceph-mon[92215]: pgmap v109: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:56 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:27:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:56 smithi026 ceph-mon[92215]: osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097] boot 2024-04-16T07:27:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:56 smithi026 ceph-mon[92215]: osdmap e64: 6 total, 6 up, 6 in 2024-04-16T07:27:56.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:27:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:56 smithi104 ceph-mon[81027]: pgmap v109: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:56 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:27:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:56 smithi104 ceph-mon[81027]: osd.2 [v2:172.21.15.26:6818/466625097,v1:172.21.15.26:6819/466625097] boot 2024-04-16T07:27:56.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:56 smithi104 ceph-mon[81027]: osdmap e64: 6 total, 6 up, 6 in 2024-04-16T07:27:56.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T07:27:57.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:57.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:57.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:57.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:57.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:57.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:57.084+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:57.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:57.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:57.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:57.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:57.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:57.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:57.372+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:57 smithi026 ceph-mon[92215]: osdmap e65: 6 total, 6 up, 6 in 2024-04-16T07:27:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:57.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:57 smithi104 ceph-mon[81027]: osdmap e65: 6 total, 6 up, 6 in 2024-04-16T07:27:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:57.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.084+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.084+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.084+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.084+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.085+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:58.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:58.085+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:58.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:58.373+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:59.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:58 smithi026 ceph-mon[92215]: pgmap v112: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:27:59.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:27:59.085+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:27:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:58 smithi104 ceph-mon[81027]: pgmap v112: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:27:59.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:27:59.372+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:00.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:00.085+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:00.208 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:59 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:00.208 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:27:59 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:00.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:59 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:00.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:27:59 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:00.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:00.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:00.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:00.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:00.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:00.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:00.373+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:01.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:00 smithi026 ceph-mon[92215]: pgmap v113: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:28:01.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:01.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:01.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:01.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:01.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:01.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:01.085+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:01.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:00 smithi104 ceph-mon[81027]: pgmap v113: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 584 KiB data, 973 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:01.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:01.373+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:02.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:02.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:02.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:02.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:02.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:02.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:02.085+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:01 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded) 2024-04-16T07:28:02.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:01 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:28:02.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:01 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded) 2024-04-16T07:28:02.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:01 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:02.373+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 146 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:03.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.086+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:03.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114178 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 298, success, outsize: 120 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.122+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.122+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.122+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.122+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.122+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114275 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.158+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.158+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 300, success, outsize: 120 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.158+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.159+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.159+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114375 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 302, success, outsize: 120 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.192+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.192+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.192+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.192+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.192+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114481 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 304, success, outsize: 120 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.222+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.222+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.222+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.222+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.222+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114584 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 306, success, outsize: 120 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.253+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.253+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.253+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.253+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.253+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:02 smithi026 ceph-mon[92215]: pgmap v114: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:03.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114681 2024-04-16T07:28:03.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 308, success, outsize: 120 2024-04-16T07:28:03.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.283+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.283+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.283+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.283+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.283+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114781 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 310, success, outsize: 120 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.312+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.312+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.312+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.312+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.312+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:02 smithi104 ceph-mon[81027]: pgmap v114: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114887 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 312, success, outsize: 120 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.340+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.340+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.340+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.340+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.340+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114990 2024-04-16T07:28:03.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 314, success, outsize: 120 2024-04-16T07:28:03.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.362+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.362+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.362+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.362+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.362+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:03.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:03.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:03.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115087 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 316, success, outsize: 120 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.384+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.385+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.385+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.385+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.385+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115187 2024-04-16T07:28:03.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 318, success, outsize: 120 2024-04-16T07:28:03.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.410+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.410+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.410+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.410+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.410+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115293 2024-04-16T07:28:03.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 320, success, outsize: 120 2024-04-16T07:28:03.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.437+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.437+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.437+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.437+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.437+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115396 2024-04-16T07:28:03.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 322, success, outsize: 120 2024-04-16T07:28:03.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.465+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.465+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.465+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.465+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.465+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115493 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 324, success, outsize: 120 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.490+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.490+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.490+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.490+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.490+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115593 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 326, success, outsize: 120 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.519+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.519+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.519+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.519+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.519+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115699 2024-04-16T07:28:03.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 328, success, outsize: 120 2024-04-16T07:28:03.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.549+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.549+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.549+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.549+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.549+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115802 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 330, success, outsize: 120 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.577+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.577+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.577+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.577+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.577+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115899 2024-04-16T07:28:03.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 332, success, outsize: 120 2024-04-16T07:28:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.605+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.605+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.605+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.605+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.605+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115999 2024-04-16T07:28:03.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 334, success, outsize: 120 2024-04-16T07:28:03.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.634+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.634+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.634+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.634+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.634+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116105 2024-04-16T07:28:03.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 336, success, outsize: 120 2024-04-16T07:28:03.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.668+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.668+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.668+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.668+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.668+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116208 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 338, success, outsize: 120 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.701+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.701+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.701+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.701+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.701+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116305 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 340, success, outsize: 120 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.733+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.733+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.733+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.733+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.733+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116405 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 342, success, outsize: 120 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.768+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.768+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.768+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.768+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.768+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116511 2024-04-16T07:28:03.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 344, success, outsize: 120 2024-04-16T07:28:03.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.807+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.807+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.807+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.807+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.807+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116614 2024-04-16T07:28:03.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 346, success, outsize: 120 2024-04-16T07:28:03.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.840+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.840+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.840+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.840+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.840+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116711 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 348, success, outsize: 120 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.874+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.874+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.874+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.874+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.874+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116811 2024-04-16T07:28:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 350, success, outsize: 120 2024-04-16T07:28:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.911+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.911+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.911+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.911+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.911+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:03.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116917 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 352, success, outsize: 120 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.952+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.952+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.952+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.952+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:28:03.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:03.952+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:28:04.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:04.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:28:04.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:28:04.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 25) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 30 ==== client_session(renewcaps seq 25) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:28:04.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:04.087+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 25) v1 from mds.0 2024-04-16T07:28:04.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:04.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:04.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:04.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:04.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:04.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:04.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:05.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:05.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:05.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:05.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:05.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:05.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:05.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: pgmap v115: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:05.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:28:05.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: pgmap v115: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:28:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:05.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:05.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:05.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:05.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:05.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:05.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:05.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.086+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.086+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.086+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.086+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:06.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:06.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:06.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:06.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.233 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T07:28:06.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:06.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:06.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:06.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:06.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:07.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:07.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:07.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:07.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:07.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:07.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:07.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:07.244 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:06 smithi104 ceph-mon[81027]: pgmap v116: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:07.244 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:06 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T07:28:07.245 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:06 smithi104 ceph-mon[81027]: Upgrade: osd.3 is safe to restart 2024-04-16T07:28:07.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:06 smithi026 ceph-mon[92215]: pgmap v116: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:07.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:06 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T07:28:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:06 smithi026 ceph-mon[92215]: Upgrade: osd.3 is safe to restart 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:07.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:07.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:08.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:08.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:08.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:08.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:08.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:08.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:08.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:08.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:08.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:08.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:08.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:08.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:08.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:08.374+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:08.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: pgmap v117: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:08.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: Upgrade: Updating osd.3 2024-04-16T07:28:08.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:08.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:28:08.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:08.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:08 smithi104 ceph-mon[81027]: Deploying daemon osd.3 on smithi104 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: pgmap v117: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: Upgrade: Updating osd.3 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:08 smithi026 ceph-mon[92215]: Deploying daemon osd.3 on smithi104 2024-04-16T07:28:09.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:09.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:09.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:09.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:09.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:09.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:09.087+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 -- 172.21.15.26:0/3367303993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 msgr2=0x7f001c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 --2- 172.21.15.26:0/3367303993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c070460 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0010004940 tx=0x7f0010009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 -- 172.21.15.26:0/3367303993 shutdown_connections 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 --2- 172.21.15.26:0/3367303993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 0x7f001c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 --2- 172.21.15.26:0/3367303993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c070460 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 -- 172.21.15.26:0/3367303993 >> 172.21.15.26:0/3367303993 conn(0x7f001c0f8530 msgr2=0x7f001c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 -- 172.21.15.26:0/3367303993 shutdown_connections 2024-04-16T07:28:09.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.347+0000 7f0024200700 1 -- 172.21.15.26:0/3367303993 wait complete. 2024-04-16T07:28:09.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.348+0000 7f0024200700 1 Processor -- start 2024-04-16T07:28:09.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0024200700 1 -- start start 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0024200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 0x7f001c10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0024200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0024200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c110870 con 0x7f001c071f50 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0024200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c110fb0 con 0x7f001c0709a0 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0020e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0020e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60484/0 (socket says 172.21.15.26:60484) 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.349+0000 7f0020e00700 1 -- 172.21.15.26:0/1153124902 learned_addr learned my addr 172.21.15.26:0/1153124902 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.350+0000 7f0021800700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 0x7f001c10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.350+0000 7f0020e00700 1 -- 172.21.15.26:0/1153124902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 msgr2=0x7f001c10fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.350+0000 7f0020e00700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 0x7f001c10fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.350+0000 7f0020e00700 1 -- 172.21.15.26:0/1153124902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010005040 con 0x7f001c071f50 2024-04-16T07:28:09.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.350+0000 7f0020e00700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f000800c830 tx=0x7f0008004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:09.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.351+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000800a760 con 0x7f001c071f50 2024-04-16T07:28:09.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.351+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f000800ad90 con 0x7f001c071f50 2024-04-16T07:28:09.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.351+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c111290 con 0x7f001c071f50 2024-04-16T07:28:09.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.351+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00080113a0 con 0x7f001c071f50 2024-04-16T07:28:09.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.351+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c111850 con 0x7f001c071f50 2024-04-16T07:28:09.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.352+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c05f5f0 con 0x7f001c071f50 2024-04-16T07:28:09.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.353+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0008012400 con 0x7f001c071f50 2024-04-16T07:28:09.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.354+0000 7f0019600700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 0x7f00040710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.354+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f000808e970 con 0x7f001c071f50 2024-04-16T07:28:09.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.355+0000 7f0021800700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 0x7f00040710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.355+0000 7f0021800700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 0x7f00040710c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0010009fd0 tx=0x7f0010009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:09.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.360+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f000805b250 con 0x7f001c071f50 2024-04-16T07:28:09.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:09.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:09.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:09.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:09.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:09.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:09.375+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:09.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.615+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f001c111c50 con 0x7f000406ec10 2024-04-16T07:28:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.618+0000 7f0019600700 1 -- 172.21.15.26:0/1153124902 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f001c111c50 con 0x7f000406ec10 2024-04-16T07:28:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 msgr2=0x7f00040710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 0x7f00040710c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0010009fd0 tx=0x7f0010009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 msgr2=0x7f001c110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f000800c830 tx=0x7f0008004470 comp rx=0 tx=0).stop 2024-04-16T07:28:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 shutdown_connections 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f000406ec10 0x7f00040710c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f001c0709a0 0x7f001c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 --2- 172.21.15.26:0/1153124902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f001c071f50 0x7f001c110270 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 >> 172.21.15.26:0/1153124902 conn(0x7f001c0f8530 msgr2=0x7f001c0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 shutdown_connections 2024-04-16T07:28:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.621+0000 7f0024200700 1 -- 172.21.15.26:0/1153124902 wait complete. 2024-04-16T07:28:09.636 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.783+0000 7f36ccc00700 1 -- 172.21.15.26:0/774442236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c80ff020 msgr2=0x7f36c80ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.783+0000 7f36ccc00700 1 --2- 172.21.15.26:0/774442236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c80ff020 0x7f36c80ff430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36b8004940 tx=0x7f36b8009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 -- 172.21.15.26:0/774442236 shutdown_connections 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 --2- 172.21.15.26:0/774442236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c81002d0 0x7f36c8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 --2- 172.21.15.26:0/774442236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c80ff020 0x7f36c80ff430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 -- 172.21.15.26:0/774442236 >> 172.21.15.26:0/774442236 conn(0x7f36c80fa950 msgr2=0x7f36c80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 -- 172.21.15.26:0/774442236 shutdown_connections 2024-04-16T07:28:09.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.784+0000 7f36ccc00700 1 -- 172.21.15.26:0/774442236 wait complete. 2024-04-16T07:28:09.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 Processor -- start 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 -- start start 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c80ff020 0x7f36c810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c81106a0 con 0x7f36c81002d0 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.785+0000 7f36ccc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8110810 con 0x7f36c80ff020 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60504/0 (socket says 172.21.15.26:60504) 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 -- 172.21.15.26:0/2120438232 learned_addr learned my addr 172.21.15.26:0/2120438232 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5e00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c80ff020 0x7f36c810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 -- 172.21.15.26:0/2120438232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c80ff020 msgr2=0x7f36c810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c80ff020 0x7f36c810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.786+0000 7f36c5400700 1 -- 172.21.15.26:0/2120438232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b8005040 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36c5400700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36b4002a50 tx=0x7f36b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b4007cc0 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36b400f040 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b400da30 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c8115270 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.787+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8115830 con 0x7f36c81002d0 2024-04-16T07:28:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.788+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c804c710 con 0x7f36c81002d0 2024-04-16T07:28:09.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.789+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f36b400db90 con 0x7f36c81002d0 2024-04-16T07:28:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.791+0000 7f36be000700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 0x7f36ac0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.791+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f36b408e5d0 con 0x7f36c81002d0 2024-04-16T07:28:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.791+0000 7f36c5e00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 0x7f36ac0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:09.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.792+0000 7f36c5e00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 0x7f36ac0710b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36b800b3b0 tx=0x7f36b8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:09.796+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f36b405aeb0 con 0x7f36c81002d0 2024-04-16T07:28:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.044+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36c8104690 con 0x7f36ac06ec00 2024-04-16T07:28:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.045+0000 7f36be000700 1 -- 172.21.15.26:0/2120438232 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f36c8104690 con 0x7f36ac06ec00 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.048+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 msgr2=0x7f36ac0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.048+0000 7f36ccc00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 0x7f36ac0710b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36b800b3b0 tx=0x7f36b8009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.048+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 msgr2=0x7f36c81100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.048+0000 7f36ccc00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36b4002a50 tx=0x7f36b40046b0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 shutdown_connections 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f36ac06ec00 0x7f36ac0710b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36c80ff020 0x7f36c810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 --2- 172.21.15.26:0/2120438232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36c81002d0 0x7f36c81100a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 >> 172.21.15.26:0/2120438232 conn(0x7f36c80fa950 msgr2=0x7f36c80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 shutdown_connections 2024-04-16T07:28:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.049+0000 7f36ccc00700 1 -- 172.21.15.26:0/2120438232 wait complete. 2024-04-16T07:28:10.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.087+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:10.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.087+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:10.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.087+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:10.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.087+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:10.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.088+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:10.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:10.088+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 -- 172.21.15.26:0/515289104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 msgr2=0x7fedac100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 --2- 172.21.15.26:0/515289104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac100740 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fed98004940 tx=0x7fed98009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 -- 172.21.15.26:0/515289104 shutdown_connections 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 --2- 172.21.15.26:0/515289104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac100740 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 --2- 172.21.15.26:0/515289104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedac0ff020 0x7fedac0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 -- 172.21.15.26:0/515289104 >> 172.21.15.26:0/515289104 conn(0x7fedac0fa950 msgr2=0x7fedac0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 -- 172.21.15.26:0/515289104 shutdown_connections 2024-04-16T07:28:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.197+0000 7fedb1400700 1 -- 172.21.15.26:0/515289104 wait complete. 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.198+0000 7fedb1400700 1 Processor -- start 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.198+0000 7fedb1400700 1 -- start start 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7fedb1400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedac0ff020 0x7fedac10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7fedb1400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7fedb1400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac1106b0 con 0x7fedac1002d0 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7fedb1400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac110820 con 0x7fedac0ff020 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60522/0 (socket says 172.21.15.26:60522) 2024-04-16T07:28:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 -- 172.21.15.26:0/3545628839 learned_addr learned my addr 172.21.15.26:0/3545628839 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 -- 172.21.15.26:0/3545628839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedac0ff020 msgr2=0x7fedac10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedac0ff020 0x7fedac10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.199+0000 7feda9e00700 1 -- 172.21.15.26:0/3545628839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed98005040 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7feda9e00700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed98009150 tx=0x7fed98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed9800e040 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed9800bcc0 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed9801c630 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac115220 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.200+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac115790 con 0x7fedac1002d0 2024-04-16T07:28:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.201+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac04c710 con 0x7fedac1002d0 2024-04-16T07:28:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.202+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fed98013420 con 0x7fedac1002d0 2024-04-16T07:28:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.203+0000 7feda2a00700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 0x7fed900710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.203+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed9808f4f0 con 0x7fedac1002d0 2024-04-16T07:28:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.204+0000 7fedaa800700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 0x7fed900710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.204+0000 7fedaa800700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 0x7fed900710b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed9c004d00 tx=0x7fed9c009ce0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.208+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fed9805be50 con 0x7fedac1002d0 2024-04-16T07:28:10.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:10.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:10.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:10.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:10.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.375+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:10.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:10.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.457+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fedac1045f0 con 0x7fed9006ec00 2024-04-16T07:28:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.487+0000 7feda2a00700 1 -- 172.21.15.26:0/3545628839 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fedac1045f0 con 0x7fed9006ec00 2024-04-16T07:28:10.487 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:28:10.487 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (5m) 13s ago 15m 13.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (2m) 13s ago 15m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (112s) 108s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (5m) 13s ago 15m 72.2M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (8m) 13s ago 8m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (8m) 13s ago 8m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (8m) 108s ago 8m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (8m) 108s ago 8m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (4m) 13s ago 17m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (3m) 108s ago 14m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (3m) 13s ago 17m 45.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (2m) 108s ago 14m 30.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (5m) 13s ago 15m 7931k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (4m) 108s ago 14m 7578k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (81s) 13s ago 13m 48.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (51s) 13s ago 12m 48.4M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (21s) 13s ago 11m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:28:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (11m) 108s ago 11m 46.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 37b176631443 2024-04-16T07:28:10.489 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (10m) 108s ago 10m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:28:10.489 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (10m) 108s ago 10m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:28:10.489 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (3m) 13s ago 14m 57.6M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:28:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 msgr2=0x7fed900710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 0x7fed900710b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed9c004d00 tx=0x7fed9c009ce0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 msgr2=0x7fedac1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed98009150 tx=0x7fed98009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 shutdown_connections 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fed9006ec00 0x7fed900710b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedac0ff020 0x7fedac10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 --2- 172.21.15.26:0/3545628839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fedac1002d0 0x7fedac1100b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.490+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 >> 172.21.15.26:0/3545628839 conn(0x7fedac0fa950 msgr2=0x7fedac0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.491+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 shutdown_connections 2024-04-16T07:28:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.491+0000 7fedb1400700 1 -- 172.21.15.26:0/3545628839 wait complete. 2024-04-16T07:28:10.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:10 smithi104 ceph-mon[81027]: pgmap v118: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:10.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:10.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:10 smithi104 ceph-mon[81027]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 -- 172.21.15.26:0/2664955988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80662f0 msgr2=0x7f8bd80ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 --2- 172.21.15.26:0/2664955988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80662f0 0x7f8bd80ffa30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8004940 tx=0x7f8bc8009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 -- 172.21.15.26:0/2664955988 shutdown_connections 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 --2- 172.21.15.26:0/2664955988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80fff70 0x7f8bd8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 --2- 172.21.15.26:0/2664955988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80662f0 0x7f8bd80ffa30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 -- 172.21.15.26:0/2664955988 >> 172.21.15.26:0/2664955988 conn(0x7f8bd80f74c0 msgr2=0x7f8bd80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 -- 172.21.15.26:0/2664955988 shutdown_connections 2024-04-16T07:28:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.645+0000 7f8bdd800700 1 -- 172.21.15.26:0/2664955988 wait complete. 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.646+0000 7f8bdd800700 1 Processor -- start 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.646+0000 7f8bdd800700 1 -- start start 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bdd800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80662f0 0x7f8bd810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bdd800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bdd800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8110790 con 0x7f8bd80fff70 2024-04-16T07:28:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bdd800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8110f90 con 0x7f8bd80662f0 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bd6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bd6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60534/0 (socket says 172.21.15.26:60534) 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bd6000700 1 -- 172.21.15.26:0/1415376088 learned_addr learned my addr 172.21.15.26:0/1415376088 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.647+0000 7f8bd6a00700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80662f0 0x7f8bd810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bd6000700 1 -- 172.21.15.26:0/1415376088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80662f0 msgr2=0x7f8bd810fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bd6000700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80662f0 0x7f8bd810fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bd6000700 1 -- 172.21.15.26:0/1415376088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8005040 con 0x7f8bd80fff70 2024-04-16T07:28:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bd6000700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400eb40 tx=0x7f8bc400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bc4007cc0 con 0x7f8bd80fff70 2024-04-16T07:28:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bc400f040 con 0x7f8bd80fff70 2024-04-16T07:28:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bc400ca10 con 0x7f8bd80fff70 2024-04-16T07:28:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd8111210 con 0x7f8bd80fff70 2024-04-16T07:28:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.648+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd8111750 con 0x7f8bd80fff70 2024-04-16T07:28:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.649+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd805f5f0 con 0x7f8bd80fff70 2024-04-16T07:28:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.651+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8bc400cb70 con 0x7f8bd80fff70 2024-04-16T07:28:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.652+0000 7f8bcea00700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 0x7f8bbc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.652+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8bc408e3f0 con 0x7f8bd80fff70 2024-04-16T07:28:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.652+0000 7f8bd6a00700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 0x7f8bbc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.653+0000 7f8bd6a00700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 0x7f8bbc0710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8013870 tx=0x7f8bc8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.657+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8bc405ad80 con 0x7f8bd80fff70 2024-04-16T07:28:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:10 smithi026 ceph-mon[92215]: pgmap v118: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:10.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:10 smithi026 ceph-mon[92215]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.985+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8bd804bed0 con 0x7f8bd80fff70 2024-04-16T07:28:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.986+0000 7f8bcea00700 1 -- 172.21.15.26:0/1415376088 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f8bc405a910 con 0x7f8bd80fff70 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:28:10.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 7, 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 7 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:28:10.987 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.989+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 msgr2=0x7f8bbc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.989+0000 7f8bdd800700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 0x7f8bbc0710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8013870 tx=0x7f8bc8009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.989+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 msgr2=0x7f8bd8110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.989+0000 7f8bdd800700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400eb40 tx=0x7f8bc400b040 comp rx=0 tx=0).stop 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 shutdown_connections 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f8bbc06ec10 0x7f8bbc0710c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bd80662f0 0x7f8bd810fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 --2- 172.21.15.26:0/1415376088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd80fff70 0x7f8bd8110190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 >> 172.21.15.26:0/1415376088 conn(0x7f8bd80f74c0 msgr2=0x7f8bd80fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:10.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 shutdown_connections 2024-04-16T07:28:10.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:10.990+0000 7f8bdd800700 1 -- 172.21.15.26:0/1415376088 wait complete. 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:11.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:11.088+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 -- 172.21.15.26:0/2919523430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 msgr2=0x7f2680103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 --2- 172.21.15.26:0/2919523430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680103830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2674009ce0 tx=0x7f2674004d10 comp rx=0 tx=0).stop 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 -- 172.21.15.26:0/2919523430 shutdown_connections 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 --2- 172.21.15.26:0/2919523430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 0x7f2680104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 --2- 172.21.15.26:0/2919523430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680103830 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.151+0000 7f2688200700 1 -- 172.21.15.26:0/2919523430 >> 172.21.15.26:0/2919523430 conn(0x7f2680072640 msgr2=0x7f2680074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.152+0000 7f2688200700 1 -- 172.21.15.26:0/2919523430 shutdown_connections 2024-04-16T07:28:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.152+0000 7f2688200700 1 -- 172.21.15.26:0/2919523430 wait complete. 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.152+0000 7f2688200700 1 Processor -- start 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2688200700 1 -- start start 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2688200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2688200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 0x7f2680114650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2688200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680114c50 con 0x7f2680103420 2024-04-16T07:28:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2688200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680114dc0 con 0x7f26801046d0 2024-04-16T07:28:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2685800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2685800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60556/0 (socket says 172.21.15.26:60556) 2024-04-16T07:28:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.153+0000 7f2685800700 1 -- 172.21.15.26:0/3222859705 learned_addr learned my addr 172.21.15.26:0/3222859705 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.154+0000 7f2684e00700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 0x7f2680114650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.154+0000 7f2685800700 1 -- 172.21.15.26:0/3222859705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 msgr2=0x7f2680114650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.154+0000 7f2685800700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 0x7f2680114650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.154+0000 7f2685800700 1 -- 172.21.15.26:0/3222859705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674009940 con 0x7f2680103420 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.154+0000 7f2685800700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2674005010 tx=0x7f2674004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.155+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267400e040 con 0x7f2680103420 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.155+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2674008ed0 con 0x7f2680103420 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.155+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26801197c0 con 0x7f2680103420 2024-04-16T07:28:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.155+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267401c870 con 0x7f2680103420 2024-04-16T07:28:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.155+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2680119d80 con 0x7f2680103420 2024-04-16T07:28:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.156+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268004c710 con 0x7f2680103420 2024-04-16T07:28:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.157+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2674012c70 con 0x7f2680103420 2024-04-16T07:28:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.158+0000 7f267d600700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 0x7f26680710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.158+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2674090530 con 0x7f2680103420 2024-04-16T07:28:11.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.159+0000 7f2684e00700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 0x7f26680710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.159+0000 7f2684e00700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 0x7f26680710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f266c004990 tx=0x7f266c009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:11.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.163+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f267405cd90 con 0x7f2680103420 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:11.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:11.375+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:11.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.450+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2680119fa0 con 0x7f2680103420 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.450+0000 7f267d600700 1 -- 172.21.15.26:0/3222859705 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f2680119fa0 con 0x7f2680103420 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:28:11.450 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:28:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:11.452 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:11.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 msgr2=0x7f26680710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 0x7f26680710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f266c004990 tx=0x7f266c009ee0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 msgr2=0x7f2680114110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2674005010 tx=0x7f2674004d10 comp rx=0 tx=0).stop 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 shutdown_connections 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f266806ec00 0x7f26680710b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2680103420 0x7f2680114110 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 --2- 172.21.15.26:0/3222859705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26801046d0 0x7f2680114650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 >> 172.21.15.26:0/3222859705 conn(0x7f2680072640 msgr2=0x7f2680073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 shutdown_connections 2024-04-16T07:28:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.454+0000 7f2688200700 1 -- 172.21.15.26:0/3222859705 wait complete. 2024-04-16T07:28:11.457 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:28:11.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:11 smithi104 ceph-mon[81027]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:11.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:11 smithi104 ceph-mon[81027]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:11.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:11 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1415376088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:11.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 -- 172.21.15.26:0/1615816309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d01002d0 msgr2=0x7fc4d0100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 --2- 172.21.15.26:0/1615816309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d01002d0 0x7fc4d0100740 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc004940 tx=0x7fc4bc009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 -- 172.21.15.26:0/1615816309 shutdown_connections 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 --2- 172.21.15.26:0/1615816309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d01002d0 0x7fc4d0100740 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 --2- 172.21.15.26:0/1615816309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d00ff020 0x7fc4d00ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 -- 172.21.15.26:0/1615816309 >> 172.21.15.26:0/1615816309 conn(0x7fc4d00fa950 msgr2=0x7fc4d00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 -- 172.21.15.26:0/1615816309 shutdown_connections 2024-04-16T07:28:11.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.604+0000 7fc4d6000700 1 -- 172.21.15.26:0/1615816309 wait complete. 2024-04-16T07:28:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.605+0000 7fc4d6000700 1 Processor -- start 2024-04-16T07:28:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4d6000700 1 -- start start 2024-04-16T07:28:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4d6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4d6000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d01002d0 0x7fc4d01100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4d6000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d01106b0 con 0x7fc4d00ff020 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4d6000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d0110820 con 0x7fc4d01002d0 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4cf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4cf400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60574/0 (socket says 172.21.15.26:60574) 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.606+0000 7fc4cf400700 1 -- 172.21.15.26:0/257589421 learned_addr learned my addr 172.21.15.26:0/257589421 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4cea00700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d01002d0 0x7fc4d01100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4cf400700 1 -- 172.21.15.26:0/257589421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d01002d0 msgr2=0x7fc4d01100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4cf400700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d01002d0 0x7fc4d01100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4cf400700 1 -- 172.21.15.26:0/257589421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc005040 con 0x7fc4d00ff020 2024-04-16T07:28:11.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4cf400700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000c8c0 tx=0x7fc4c00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c000a980 con 0x7fc4d00ff020 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4c000ce60 con 0x7fc4d00ff020 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c00115e0 con 0x7fc4d00ff020 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.607+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d0115280 con 0x7fc4d00ff020 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.608+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d0115840 con 0x7fc4d00ff020 2024-04-16T07:28:11.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.609+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d004c710 con 0x7fc4d00ff020 2024-04-16T07:28:11.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.610+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc4c0012400 con 0x7fc4d00ff020 2024-04-16T07:28:11.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.611+0000 7fc4c7400700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 0x7fc4b40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:11.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.611+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4c008e910 con 0x7fc4d00ff020 2024-04-16T07:28:11.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.611+0000 7fc4cea00700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 0x7fc4b40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:11.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.612+0000 7fc4cea00700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 0x7fc4b40710b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc004f90 tx=0x7fc4bc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:11.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.616+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc4c005b270 con 0x7fc4d00ff020 2024-04-16T07:28:11.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:11 smithi026 ceph-mon[92215]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:11.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:11 smithi026 ceph-mon[92215]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:11.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:11 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1415376088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:11.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.869+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4d01045f0 con 0x7fc4b406ec00 2024-04-16T07:28:11.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.870+0000 7fc4c7400700 1 -- 172.21.15.26:0/257589421 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc4d01045f0 con 0x7fc4b406ec00 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:28:11.870 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:28:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:28:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "9/21 daemons upgraded", 2024-04-16T07:28:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T07:28:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:28:11.871 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:28:11.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 msgr2=0x7fc4b40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 0x7fc4b40710b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc004f90 tx=0x7fc4bc009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 msgr2=0x7fc4d010fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000c8c0 tx=0x7fc4c00044c0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 shutdown_connections 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc4b406ec00 0x7fc4b40710b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4d00ff020 0x7fc4d010fb70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 --2- 172.21.15.26:0/257589421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4d01002d0 0x7fc4d01100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 >> 172.21.15.26:0/257589421 conn(0x7fc4d00fa950 msgr2=0x7fc4d00fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 shutdown_connections 2024-04-16T07:28:11.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:11.873+0000 7fc4d6000700 1 -- 172.21.15.26:0/257589421 wait complete. 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 -- 172.21.15.26:0/3508952577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 msgr2=0x7f169c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 --2- 172.21.15.26:0/3508952577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c070d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1688004970 tx=0x7f1688009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 -- 172.21.15.26:0/3508952577 shutdown_connections 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 --2- 172.21.15.26:0/3508952577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c070d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 --2- 172.21.15.26:0/3508952577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.030+0000 7f169b200700 1 -- 172.21.15.26:0/3508952577 >> 172.21.15.26:0/3508952577 conn(0x7f169c0f8390 msgr2=0x7f169c0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:12.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.031+0000 7f169b200700 1 -- 172.21.15.26:0/3508952577 shutdown_connections 2024-04-16T07:28:12.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.031+0000 7f169b200700 1 -- 172.21.15.26:0/3508952577 wait complete. 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.031+0000 7f169b200700 1 Processor -- start 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f169b200700 1 -- start start 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f169b200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c10f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f169b200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f169b200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c110520 con 0x7f169c071e40 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f169b200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c110d70 con 0x7f169c070890 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f1699400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f1699e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c10f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f1699400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60584/0 (socket says 172.21.15.26:60584) 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.032+0000 7f1699400700 1 -- 172.21.15.26:0/4142992748 learned_addr learned my addr 172.21.15.26:0/4142992748 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.033+0000 7f1699400700 1 -- 172.21.15.26:0/4142992748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 msgr2=0x7f169c10f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:12.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.033+0000 7f1699400700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c10f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.033+0000 7f1699400700 1 -- 172.21.15.26:0/4142992748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688005040 con 0x7f169c071e40 2024-04-16T07:28:12.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.033+0000 7f1699400700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1688007f50 tx=0x7f1688009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:12.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.033+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168800e040 con 0x7f169c071e40 2024-04-16T07:28:12.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.034+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1688013420 con 0x7f169c071e40 2024-04-16T07:28:12.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.034+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c110ff0 con 0x7f169c071e40 2024-04-16T07:28:12.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.034+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168801c610 con 0x7f169c071e40 2024-04-16T07:28:12.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.034+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c111530 con 0x7f169c071e40 2024-04-16T07:28:12.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.035+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c05f5f0 con 0x7f169c071e40 2024-04-16T07:28:12.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.036+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1688013a70 con 0x7f169c071e40 2024-04-16T07:28:12.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.037+0000 7f1692000700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 0x7f16800710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:12.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.037+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f168808f9a0 con 0x7f169c071e40 2024-04-16T07:28:12.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.037+0000 7f1699e00700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 0x7f16800710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:12.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.038+0000 7f1699e00700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 0x7f16800710c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f168c009b80 tx=0x7f168c00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:12.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.042+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f168805c470 con 0x7f169c071e40 2024-04-16T07:28:12.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:12.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:12.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:12.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:12.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:12.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:12.088+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:12.333 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:12 smithi104 systemd[1]: Stopping Ceph osd.3 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:28:12.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.356+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f169c04bed0 con 0x7f169c071e40 2024-04-16T07:28:12.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.357+0000 7f1692000700 1 -- 172.21.15.26:0/4142992748 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f168805c000 con 0x7f169c071e40 2024-04-16T07:28:12.356 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:28:12.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.359+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 msgr2=0x7f16800710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:12.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.359+0000 7f169b200700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 0x7f16800710c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f168c009b80 tx=0x7f168c00b040 comp rx=0 tx=0).stop 2024-04-16T07:28:12.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.359+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 msgr2=0x7f169c10ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1688007f50 tx=0x7f1688009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 shutdown_connections 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f168006ec10 0x7f16800710c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c070890 0x7f169c10f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 --2- 172.21.15.26:0/4142992748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f169c071e40 0x7f169c10ff20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 >> 172.21.15.26:0/4142992748 conn(0x7f169c0f8390 msgr2=0x7f169c0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 shutdown_connections 2024-04-16T07:28:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:12.360+0000 7f169b200700 1 -- 172.21.15.26:0/4142992748 wait complete. 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 25) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:28:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:12.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.375+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:12.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.376+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 29 ==== client_session(renewcaps seq 25) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:12.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:12.376+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 25) v1 from mds.0 2024-04-16T07:28:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:12 smithi026 ceph-mon[92215]: pgmap v119: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:12 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3222859705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:28:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:12 smithi026 ceph-mon[92215]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:12 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4142992748' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:28:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-mon[81027]: pgmap v119: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3222859705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:28:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-mon[81027]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4142992748' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:28:12.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[41629]: 2024-04-16T07:28:12.486+0000 7fc267800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:28:12.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[41629]: 2024-04-16T07:28:12.486+0000 7fc267800700 -1 osd.3 65 *** Got signal Terminated *** 2024-04-16T07:28:12.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:12 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[41629]: 2024-04-16T07:28:12.486+0000 7fc267800700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:28:13.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:13.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:13.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:13.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:13.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:13.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:13.088+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:13.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:13.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:13.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:13.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:13.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:13.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:13.375+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:13.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:13 smithi026 ceph-mon[92215]: osd.3 marked itself down and dead 2024-04-16T07:28:13.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:13 smithi104 ceph-mon[81027]: osd.3 marked itself down and dead 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:14.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:14.089+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:14.326 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:14 smithi104 podman[89757]: 2024-04-16 07:28:14.010620195 +0000 UTC m=+1.687847847 container died 37b176631443de72801a3b3db005f34b62e0481a2791666dbfb1ec53f4ca2f1d (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:28:14.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:14.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:14.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:14.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.375+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:14.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.376+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:14.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:14.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:14.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:14 smithi104 ceph-mon[81027]: pgmap v120: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:14.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:14 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:28:14.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:14 smithi104 ceph-mon[81027]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T07:28:14.583 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:14 smithi104 podman[89757]: 2024-04-16 07:28:14.455337197 +0000 UTC m=+2.132564864 container cleanup 37b176631443de72801a3b3db005f34b62e0481a2791666dbfb1ec53f4ca2f1d (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:28:14.584 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:14 smithi104 bash[89757]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3 2024-04-16T07:28:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:14 smithi026 ceph-mon[92215]: pgmap v120: 65 pgs: 65 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail 2024-04-16T07:28:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:14 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:28:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:14 smithi026 ceph-mon[92215]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T07:28:15.083 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:14 smithi104 podman[89775]: 2024-04-16 07:28:14.714737319 +0000 UTC m=+0.693444509 container remove 37b176631443de72801a3b3db005f34b62e0481a2791666dbfb1ec53f4ca2f1d (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:15.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:15.089+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:15.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:15.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:15.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:15.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:15.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:15.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:15.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:15.735 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:15 smithi104 ceph-mon[81027]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T07:28:15.736 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:15 smithi104 podman[89868]: 2024-04-16 07:28:15.426459476 +0000 UTC m=+0.033955227 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:15.736 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:15 smithi104 podman[89868]: 2024-04-16 07:28:15.557167503 +0000 UTC m=+0.164663249 container create d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:28:15.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:15 smithi026 ceph-mon[92215]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T07:28:16.082 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:15 smithi104 podman[89868]: 2024-04-16 07:28:15.848564379 +0000 UTC m=+0.456060209 container init d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T07:28:16.082 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:15 smithi104 podman[89868]: 2024-04-16 07:28:15.861701974 +0000 UTC m=+0.469197756 container start d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:28:16.082 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:15 smithi104 podman[89868]: 2024-04-16 07:28:15.919788641 +0000 UTC m=+0.527284441 container attach d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T07:28:16.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:16.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:16.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:16.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:16.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:16.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:16.089+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:16.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:16.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:16.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:16.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:16.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:16.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:16.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:16.450 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:16 smithi104 podman[89868]: 2024-04-16 07:28:16.308962937 +0000 UTC m=+0.916458725 container died d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:28:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:16 smithi026 ceph-mon[92215]: pgmap v123: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:16.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:16 smithi026 ceph-mon[92215]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-16T07:28:16.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:16 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:16.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:16 smithi104 ceph-mon[81027]: pgmap v123: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:16.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:16 smithi104 ceph-mon[81027]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-16T07:28:16.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:16 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:17.022 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:16 smithi104 podman[89895]: 2024-04-16 07:28:16.997844771 +0000 UTC m=+0.674535083 container remove d4618fd7682cec91d6a9ecd6544ca674f908dce497005f5022fe86cc5a2fc133 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-16T07:28:17.023 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-16T07:28:17.023 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 systemd[1]: Stopped Ceph osd.3 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:28:17.023 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.3.service: Consumed 13.252s CPU time. 2024-04-16T07:28:17.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:17.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:17.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:17.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:17.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:17.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:17.089+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:17.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:17.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:17.771 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 systemd[1]: Starting Ceph osd.3 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:28:18.054 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 podman[90003]: 2024-04-16 07:28:17.683916397 +0000 UTC m=+0.038359963 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:18.054 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:17 smithi104 podman[90003]: 2024-04-16 07:28:17.798818522 +0000 UTC m=+0.153262106 container create 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:28:18.054 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 podman[90003]: 2024-04-16 07:28:18.053439026 +0000 UTC m=+0.407882612 container init 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:18.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:18.090+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:18.332 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 podman[90003]: 2024-04-16 07:28:18.066954212 +0000 UTC m=+0.421397825 container start 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T07:28:18.332 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 podman[90003]: 2024-04-16 07:28:18.151325145 +0000 UTC m=+0.505768768 container attach 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-16T07:28:18.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:18.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:18.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:18.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 171 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:18.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:18.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.376+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:18.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:18 smithi026 ceph-mon[92215]: pgmap v124: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:18.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90106 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 348, success, outsize: 120 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.804+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.804+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.804+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.804+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:18.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:18.804+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-mon[81027]: pgmap v124: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:19.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:19.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:19.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:19.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:19.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:19.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:19.090+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T07:28:19.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 bash[90003]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate[90019]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T07:28:19.335 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:18 smithi104 podman[90003]: 2024-04-16 07:28:18.995247581 +0000 UTC m=+1.349691168 container died 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.375+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.375+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.378+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.378+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:19.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:19.378+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:20.082 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:19 smithi104 podman[90144]: 2024-04-16 07:28:19.804580961 +0000 UTC m=+0.789387536 container remove 6f91057c0453754cdeb654784e882981097d6aaf9c2612a1a440a8800ce1f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-16T07:28:20.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:20.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:20.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:20.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:20.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:20.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:20.090+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:20.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:20.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:20.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:20.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:20.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:20.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:20.378+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:20.454 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:20 smithi104 ceph-mon[81027]: pgmap v125: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:20.454 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:20.454 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:20.454 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 podman[90208]: 2024-04-16 07:28:20.061953239 +0000 UTC m=+0.032164012 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:20.454 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 podman[90208]: 2024-04-16 07:28:20.188748131 +0000 UTC m=+0.158958929 container create d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T07:28:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:20 smithi026 ceph-mon[92215]: pgmap v125: 65 pgs: 4 active+undersized+degraded, 20 peering, 14 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:20.832 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 podman[90208]: 2024-04-16 07:28:20.455056076 +0000 UTC m=+0.425266861 container init d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-16T07:28:20.832 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 podman[90208]: 2024-04-16 07:28:20.465713862 +0000 UTC m=+0.435924638 container start d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-16T07:28:20.832 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 ceph-osd[90227]: -- 172.21.15.104:0/2887538404 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x56132c55bba0 con 0x56132c52cc00 2024-04-16T07:28:20.832 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 ceph-osd[90227]: -- 172.21.15.104:0/2887538404 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56132c563680 con 0x56132c52cc00 2024-04-16T07:28:20.832 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 bash[90208]: d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec 2024-04-16T07:28:20.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:20 smithi104 systemd[1]: Started Ceph osd.3 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:21.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:21.091+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:21.332 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:21 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:28:21.041+0000 7f73b3a6f7c0 -1 Falling back to public interface 2024-04-16T07:28:21.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:21.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:21.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:21.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:21.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:21.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:21.377+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:21.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:21.904 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:21.905 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:22.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:22.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:22.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:22.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:22.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:22.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:22.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:22.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:22.091+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:22.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:22.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:22.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:22.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:22.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:22.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:22.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:22 smithi026 ceph-mon[92215]: pgmap v126: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:22 smithi026 ceph-mon[92215]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:22 smithi026 ceph-mon[92215]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-16T07:28:23.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:22 smithi104 ceph-mon[81027]: pgmap v126: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:23.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:22 smithi104 ceph-mon[81027]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:23.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:22 smithi104 ceph-mon[81027]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-16T07:28:23.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:23.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:23.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:23.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:23.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:23.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:23.091+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:23.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:23.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 26) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:24.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:24.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:24.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.091+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:24.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.092+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 31 ==== client_session(renewcaps seq 26) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:28:24.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:24.092+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 26) v1 from mds.0 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:24.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:24.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:24 smithi104 ceph-mon[81027]: pgmap v127: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:25.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:24 smithi026 ceph-mon[92215]: pgmap v127: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:25.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:25.092+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:25.332 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:24 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:28:24.926+0000 7f73b3a6f7c0 -1 osd.3 65 log_to_monitors true 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:25.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:25.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:26.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:25 smithi026 ceph-mon[92215]: from='osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:28:26.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:25 smithi026 ceph-mon[92215]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:28:26.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:25 smithi104 ceph-mon[81027]: from='osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:28:26.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:25 smithi104 ceph-mon[81027]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:26.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:26.092+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:26.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:26.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:26.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:26.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:26.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:26.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:26.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:26.677 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:26 smithi026 ceph-mon[92215]: pgmap v128: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:26.677 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:26 smithi026 ceph-mon[92215]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T07:28:26.677 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:26 smithi026 ceph-mon[92215]: from='osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:26.677 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:26 smithi026 ceph-mon[92215]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T07:28:26.677 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:26 smithi026 ceph-mon[92215]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:27.082 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:28:26.870+0000 7f739de00700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:28:27.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-mon[81027]: pgmap v128: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 974 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:27.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-mon[81027]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T07:28:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-mon[81027]: from='osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-mon[81027]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T07:28:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:26 smithi104 ceph-mon[81027]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.092+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.092+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.092+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.092+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.093+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:27.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:27.093+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:27.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:27.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:27.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:27.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:27.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:27.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:27.379+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:28.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:27 smithi026 ceph-mon[92215]: from='osd.3 ' entity='osd.3' 2024-04-16T07:28:28.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:27 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:28.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:27 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:28.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:27 smithi104 ceph-mon[81027]: from='osd.3 ' entity='osd.3' 2024-04-16T07:28:28.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:27 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:28.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:27 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:28.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:28.093+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:28.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:28.380+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:28 smithi104 ceph-mon[81027]: pgmap v130: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 710 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:28 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:28:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:28 smithi104 ceph-mon[81027]: osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206] boot 2024-04-16T07:28:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:28 smithi104 ceph-mon[81027]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T07:28:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:28:29.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:29.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:29.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:29.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:29.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:29.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:29.093+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:28 smithi026 ceph-mon[92215]: pgmap v130: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 710 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:28 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:28:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:28 smithi026 ceph-mon[92215]: osd.3 [v2:172.21.15.104:6800/1178751206,v1:172.21.15.104:6801/1178751206] boot 2024-04-16T07:28:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:28 smithi026 ceph-mon[92215]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T07:28:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:29.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:29.380+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:30.094+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:30.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:29 smithi026 ceph-mon[92215]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T07:28:30.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:29 smithi104 ceph-mon[81027]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T07:28:30.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:30.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:30.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:30.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 172 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:30.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:30.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.380+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:30.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 91937 2024-04-16T07:28:30.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 350, success, outsize: 120 2024-04-16T07:28:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.599+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.599+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.599+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.599+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.599+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92020 2024-04-16T07:28:30.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 352, success, outsize: 120 2024-04-16T07:28:30.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.616+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.616+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.616+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.616+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.616+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92106 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 354, success, outsize: 120 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.635+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.635+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.635+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.635+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.635+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92198 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 356, success, outsize: 120 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.658+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.658+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.658+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.658+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.658+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92287 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 358, success, outsize: 120 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.683+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.683+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.683+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.683+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.683+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92370 2024-04-16T07:28:30.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 360, success, outsize: 120 2024-04-16T07:28:30.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.706+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.706+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.706+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.706+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.706+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92456 2024-04-16T07:28:30.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 362, success, outsize: 120 2024-04-16T07:28:30.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.730+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.730+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.730+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.730+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.730+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92548 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 364, success, outsize: 120 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.750+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.750+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.750+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.750+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.750+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92637 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 366, success, outsize: 120 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.769+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.769+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.769+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.769+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.769+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92720 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 368, success, outsize: 120 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.786+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.786+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.786+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.786+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.786+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92806 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 370, success, outsize: 120 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.805+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.805+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.805+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.805+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.805+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92898 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 372, success, outsize: 120 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.826+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.826+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.826+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.826+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.826+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92987 2024-04-16T07:28:30.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 374, success, outsize: 120 2024-04-16T07:28:30.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.848+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.848+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.848+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.848+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.848+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93070 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 376, success, outsize: 120 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.868+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.868+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.868+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.868+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.868+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93156 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 378, success, outsize: 120 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.888+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.888+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.888+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.888+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.888+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93248 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 380, success, outsize: 120 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.909+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.909+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.909+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.909+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.909+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93337 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 382, success, outsize: 120 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.931+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.931+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.931+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.931+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.931+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93420 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 384, success, outsize: 120 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.951+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.951+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.951+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.951+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.951+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93506 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 386, success, outsize: 120 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.972+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.972+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.972+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.972+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.972+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:30.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93598 2024-04-16T07:28:30.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 388, success, outsize: 120 2024-04-16T07:28:30.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.994+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:30.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.994+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:30.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.994+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:30.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.994+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:30.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:30.994+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93687 2024-04-16T07:28:31.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 390, success, outsize: 120 2024-04-16T07:28:31.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.017+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.017+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.017+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.017+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.017+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93770 2024-04-16T07:28:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 392, success, outsize: 120 2024-04-16T07:28:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.038+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.039+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.039+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.039+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.039+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93856 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 394, success, outsize: 120 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.061+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.061+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.061+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.061+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.061+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93948 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 396, success, outsize: 120 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.085+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.085+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.085+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.085+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.085+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.093+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.094+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.094+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.094+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.094+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:31.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:31.094+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94037 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 398, success, outsize: 120 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.109+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.109+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.109+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.109+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.109+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94120 2024-04-16T07:28:31.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 400, success, outsize: 120 2024-04-16T07:28:31.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.132+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.132+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.132+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.132+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.132+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94206 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 402, success, outsize: 120 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.155+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.155+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.155+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.155+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.155+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94298 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 404, success, outsize: 120 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.181+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.181+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.181+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.181+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:31.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.181+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:31.190 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:30 smithi104 ceph-mon[81027]: pgmap v133: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 710 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:31.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:30 smithi026 ceph-mon[92215]: pgmap v133: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 584 KiB data, 710 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T07:28:31.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:31.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:31.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:31.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:31.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:31.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:31.380+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:32.094+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:32.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:31 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded) 2024-04-16T07:28:32.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:31 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:28:32.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:31 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded) 2024-04-16T07:28:32.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:31 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 26) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.380+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.381+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:32.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.381+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:32.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.381+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 30 ==== client_session(renewcaps seq 26) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:32.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:32.381+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 26) v1 from mds.0 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:33.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:33.094+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: pgmap v134: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T07:28:33.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:33.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:33.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:33.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:33.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:33.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:33.381+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: pgmap v134: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:33.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:33.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:34.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:34.094+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:34.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:34 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T07:28:34.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:34 smithi104 ceph-mon[81027]: Upgrade: osd.4 is safe to restart 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:34.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:34.381+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:34.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:34 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T07:28:34.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:34 smithi026 ceph-mon[92215]: Upgrade: osd.4 is safe to restart 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:35.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:35.095+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: pgmap v135: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:28:35.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:35.381+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: pgmap v135: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T07:28:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:28:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:36 smithi104 ceph-mon[81027]: Upgrade: Updating osd.4 2024-04-16T07:28:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:36 smithi104 ceph-mon[81027]: Deploying daemon osd.4 on smithi104 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:36.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:36.095+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:36.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:36.381+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:36.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:36 smithi026 ceph-mon[92215]: Upgrade: Updating osd.4 2024-04-16T07:28:36.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:36 smithi026 ceph-mon[92215]: Deploying daemon osd.4 on smithi104 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:37.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:37.095+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:37 smithi104 ceph-mon[81027]: pgmap v136: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:37.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:37.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:37.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:37.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:37.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:37.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:37.382+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:37.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:37 smithi026 ceph-mon[92215]: pgmap v136: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:38.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:38.095+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:38.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:38 smithi026 ceph-mon[92215]: pgmap v137: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:38.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:38 smithi104 ceph-mon[81027]: pgmap v137: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:38.382+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:39.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:39.096+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:39.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:39.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:39.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:39.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:39.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:39.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:39.382+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:39.449 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:39 smithi104 systemd[1]: Stopping Ceph osd.4 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:28:39.757 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[48085]: 2024-04-16T07:28:39.547+0000 7f3a10000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:28:39.758 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[48085]: 2024-04-16T07:28:39.547+0000 7f3a10000700 -1 osd.4 70 *** Got signal Terminated *** 2024-04-16T07:28:39.758 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[48085]: 2024-04-16T07:28:39.547+0000 7f3a10000700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:40.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:40.096+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:40.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:40.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:40.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:40.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:40.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:40.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:40.382+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:40 smithi026 ceph-mon[92215]: pgmap v138: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:40.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:40 smithi026 ceph-mon[92215]: osd.4 marked itself down and dead 2024-04-16T07:28:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:40 smithi104 ceph-mon[81027]: pgmap v138: 65 pgs: 65 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:40.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:40 smithi104 ceph-mon[81027]: osd.4 marked itself down and dead 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:41.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:41.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:41.313 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:41 smithi104 podman[94756]: 2024-04-16 07:28:41.027529502 +0000 UTC m=+1.637541306 container died 7133906317db0e642451f8c534ad27047f2ab1ab31b3e3bdd6aeb92e5b60d96b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409) 2024-04-16T07:28:41.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.382+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:41.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:41.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.382+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:41.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.382+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:41.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.383+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:41.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:41.383+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:41.582 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:41 smithi104 podman[94756]: 2024-04-16 07:28:41.432338025 +0000 UTC m=+2.042349862 container cleanup 7133906317db0e642451f8c534ad27047f2ab1ab31b3e3bdd6aeb92e5b60d96b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, ceph=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T07:28:41.583 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:41 smithi104 bash[94756]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4 2024-04-16T07:28:41.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:41 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:28:41.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:41 smithi104 ceph-mon[81027]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T07:28:41.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:41 smithi104 ceph-mon[81027]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T07:28:41.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:41 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:28:41.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:41 smithi026 ceph-mon[92215]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T07:28:41.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:41 smithi026 ceph-mon[92215]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T07:28:42.082 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:41 smithi104 podman[94786]: 2024-04-16 07:28:41.67540106 +0000 UTC m=+0.627229381 container remove 7133906317db0e642451f8c534ad27047f2ab1ab31b3e3bdd6aeb92e5b60d96b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True) 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:42.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:42.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:42.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:42.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:42.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:42.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:42.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:42.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:42.383+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 -- 172.21.15.26:0/1679505996 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b0071f50 msgr2=0x7fc7b0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1679505996 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b0071f50 0x7fc7b0070460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0004940 tx=0x7fc7a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 -- 172.21.15.26:0/1679505996 shutdown_connections 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1679505996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b00709a0 0x7fc7b0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1679505996 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b0071f50 0x7fc7b0070460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 -- 172.21.15.26:0/1679505996 >> 172.21.15.26:0/1679505996 conn(0x7fc7b00f8530 msgr2=0x7fc7b00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 -- 172.21.15.26:0/1679505996 shutdown_connections 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.535+0000 7fc7b5400700 1 -- 172.21.15.26:0/1679505996 wait complete. 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 Processor -- start 2024-04-16T07:28:42.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 -- start start 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0071f50 0x7fc7b0110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0110800 con 0x7fc7b00709a0 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.536+0000 7fc7b5400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0110ff0 con 0x7fc7b0071f50 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47670/0 (socket says 172.21.15.26:47670) 2024-04-16T07:28:42.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 -- 172.21.15.26:0/1508474676 learned_addr learned my addr 172.21.15.26:0/1508474676 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 -- 172.21.15.26:0/1508474676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0071f50 msgr2=0x7fc7b0110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0071f50 0x7fc7b0110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 -- 172.21.15.26:0/1508474676 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0005040 con 0x7fc7b00709a0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.537+0000 7fc7ae800700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000b3b0 tx=0x7fc7a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.538+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a000e040 con 0x7fc7b00709a0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.538+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7a000bbf0 con 0x7fc7b00709a0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.538+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b0111270 con 0x7fc7b00709a0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.538+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a001c630 con 0x7fc7b00709a0 2024-04-16T07:28:42.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.538+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b0111830 con 0x7fc7b00709a0 2024-04-16T07:28:42.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.539+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b01095a0 con 0x7fc7b00709a0 2024-04-16T07:28:42.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.540+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc7a000bd60 con 0x7fc7b00709a0 2024-04-16T07:28:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.541+0000 7fc7a6a00700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 0x7fc7940710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.541+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc7a008fbe0 con 0x7fc7b00709a0 2024-04-16T07:28:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.541+0000 7fc7a7e00700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 0x7fc7940710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.542+0000 7fc7a7e00700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 0x7fc7940710c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798006db0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:42.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.547+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc7a005c540 con 0x7fc7b00709a0 2024-04-16T07:28:42.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:42 smithi026 ceph-mon[92215]: pgmap v140: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:42.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.799+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7b0111c70 con 0x7fc79406ec10 2024-04-16T07:28:42.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.802+0000 7fc7a6a00700 1 -- 172.21.15.26:0/1508474676 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc7b0111c70 con 0x7fc79406ec10 2024-04-16T07:28:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 msgr2=0x7fc7940710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 0x7fc7940710c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798006db0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 msgr2=0x7fc7b010fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000b3b0 tx=0x7fc7a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 shutdown_connections 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc79406ec10 0x7fc7940710c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7b00709a0 0x7fc7b010fc30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 --2- 172.21.15.26:0/1508474676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0071f50 0x7fc7b0110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 >> 172.21.15.26:0/1508474676 conn(0x7fc7b00f8530 msgr2=0x7fc7b0102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:42.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.805+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 shutdown_connections 2024-04-16T07:28:42.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.806+0000 7fc7b5400700 1 -- 172.21.15.26:0/1508474676 wait complete. 2024-04-16T07:28:42.821 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:28:42.827 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:42 smithi104 ceph-mon[81027]: pgmap v140: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:42.828 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:42 smithi104 podman[94875]: 2024-04-16 07:28:42.38468979 +0000 UTC m=+0.032296862 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:42.828 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:42 smithi104 podman[94875]: 2024-04-16 07:28:42.492406129 +0000 UTC m=+0.140013234 container create 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T07:28:42.828 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:42 smithi104 podman[94875]: 2024-04-16 07:28:42.8292993 +0000 UTC m=+0.476906428 container init 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 -- 172.21.15.26:0/2206460476 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b841046d0 msgr2=0x7f4b84104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 --2- 172.21.15.26:0/2206460476 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b841046d0 0x7f4b84104b40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4b70004940 tx=0x7f4b70009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 -- 172.21.15.26:0/2206460476 shutdown_connections 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 --2- 172.21.15.26:0/2206460476 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b841046d0 0x7f4b84104b40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 --2- 172.21.15.26:0/2206460476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b84103420 0x7f4b84103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.967+0000 7f4b89400700 1 -- 172.21.15.26:0/2206460476 >> 172.21.15.26:0/2206460476 conn(0x7f4b84072640 msgr2=0x7f4b84074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.968+0000 7f4b89400700 1 -- 172.21.15.26:0/2206460476 shutdown_connections 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.968+0000 7f4b89400700 1 -- 172.21.15.26:0/2206460476 wait complete. 2024-04-16T07:28:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.968+0000 7f4b89400700 1 Processor -- start 2024-04-16T07:28:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b89400700 1 -- start start 2024-04-16T07:28:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b89400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b89400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 0x7f4b84114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b89400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84114a90 con 0x7f4b84103420 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b89400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84114c00 con 0x7f4b841046d0 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b82800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b82800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47686/0 (socket says 172.21.15.26:47686) 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.969+0000 7f4b82800700 1 -- 172.21.15.26:0/2888267691 learned_addr learned my addr 172.21.15.26:0/2888267691 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b81e00700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 0x7f4b84114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b82800700 1 -- 172.21.15.26:0/2888267691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 msgr2=0x7f4b84114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b82800700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 0x7f4b84114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b82800700 1 -- 172.21.15.26:0/2888267691 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70005040 con 0x7f4b84103420 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b81e00700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 0x7f4b84114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.970+0000 7f4b82800700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4b74002a50 tx=0x7f4b740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.971+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b74007cc0 con 0x7f4b84103420 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.971+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b7400f040 con 0x7f4b84103420 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.971+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b84119660 con 0x7f4b84103420 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.971+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b74009640 con 0x7f4b84103420 2024-04-16T07:28:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.971+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b84119c20 con 0x7f4b84103420 2024-04-16T07:28:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.972+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8404c710 con 0x7f4b84103420 2024-04-16T07:28:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.973+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b740097a0 con 0x7f4b84103420 2024-04-16T07:28:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.974+0000 7f4b7aa00700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 0x7f4b680710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.974+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4b7408e700 con 0x7f4b84103420 2024-04-16T07:28:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.974+0000 7f4b81e00700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 0x7f4b680710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.975+0000 7f4b81e00700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 0x7f4b680710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000b3b0 tx=0x7f4b70009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:42.979+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4b7405af60 con 0x7f4b84103420 2024-04-16T07:28:43.082 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:42 smithi104 podman[94875]: 2024-04-16 07:28:42.840670786 +0000 UTC m=+0.488277861 container start 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:28:43.082 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:42 smithi104 podman[94875]: 2024-04-16 07:28:42.919711393 +0000 UTC m=+0.567318505 container attach 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:43.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:43.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:43.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.234+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b84108a10 con 0x7f4b6806ec00 2024-04-16T07:28:43.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.238+0000 7f4b7aa00700 1 -- 172.21.15.26:0/2888267691 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f4b84108a10 con 0x7f4b6806ec00 2024-04-16T07:28:43.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 msgr2=0x7f4b680710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 0x7f4b680710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000b3b0 tx=0x7f4b70009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:43.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 msgr2=0x7f4b84113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4b74002a50 tx=0x7f4b740046b0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 shutdown_connections 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4b6806ec00 0x7f4b680710b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b84103420 0x7f4b84113f50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 --2- 172.21.15.26:0/2888267691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b841046d0 0x7f4b84114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 >> 172.21.15.26:0/2888267691 conn(0x7f4b84072640 msgr2=0x7f4b84073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 shutdown_connections 2024-04-16T07:28:43.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.241+0000 7f4b89400700 1 -- 172.21.15.26:0/2888267691 wait complete. 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:43.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:43.383+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 -- 172.21.15.26:0/4022661725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8070870 msgr2=0x7f5fe8070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 --2- 172.21.15.26:0/4022661725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8070870 0x7f5fe8070ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4004940 tx=0x7f5fd4009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 -- 172.21.15.26:0/4022661725 shutdown_connections 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 --2- 172.21.15.26:0/4022661725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8070870 0x7f5fe8070ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 --2- 172.21.15.26:0/4022661725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 0x7f5fe8070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.396+0000 7f5feec00700 1 -- 172.21.15.26:0/4022661725 >> 172.21.15.26:0/4022661725 conn(0x7f5fe80f8580 msgr2=0x7f5fe80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:43.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.397+0000 7f5feec00700 1 -- 172.21.15.26:0/4022661725 shutdown_connections 2024-04-16T07:28:43.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.397+0000 7f5feec00700 1 -- 172.21.15.26:0/4022661725 wait complete. 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 Processor -- start 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 -- start start 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 0x7f5fe810fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe81106e0 con 0x7f5fe8110220 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5feec00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8110820 con 0x7f5fe8071e20 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.398+0000 7f5fe7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47704/0 (socket says 172.21.15.26:47704) 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 -- 172.21.15.26:0/1670381200 learned_addr learned my addr 172.21.15.26:0/1670381200 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7e00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 0x7f5fe810fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 -- 172.21.15.26:0/1670381200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 msgr2=0x7f5fe810fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 0x7f5fe810fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 -- 172.21.15.26:0/1670381200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4005040 con 0x7f5fe8110220 2024-04-16T07:28:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.399+0000 7f5fe7400700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400b970 tx=0x7f5fd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.400+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd400e040 con 0x7f5fe8110220 2024-04-16T07:28:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.400+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fd4004290 con 0x7f5fe8110220 2024-04-16T07:28:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.400+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe8115780 con 0x7f5fe8110220 2024-04-16T07:28:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.400+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd401c3f0 con 0x7f5fe8110220 2024-04-16T07:28:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.400+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe8115cf0 con 0x7f5fe8110220 2024-04-16T07:28:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.401+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe804c710 con 0x7f5fe8110220 2024-04-16T07:28:43.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.402+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5fd400bbf0 con 0x7f5fe8110220 2024-04-16T07:28:43.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.403+0000 7f5fdfe00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 0x7f5fcc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.403+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5fd408fb90 con 0x7f5fe8110220 2024-04-16T07:28:43.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.403+0000 7f5fe7e00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 0x7f5fcc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:43.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.404+0000 7f5fe7e00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 0x7f5fcc0710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8007fd0 tx=0x7f5fd8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:43.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.408+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5fd4058940 con 0x7f5fe8110220 2024-04-16T07:28:43.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:43 smithi104 ceph-mon[81027]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:43.583 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:43 smithi104 podman[94875]: 2024-04-16 07:28:43.292894396 +0000 UTC m=+0.940501507 container died 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T07:28:43.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.654+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe8104000 con 0x7f5fcc06ec00 2024-04-16T07:28:43.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.670+0000 7f5fdfe00700 1 -- 172.21.15.26:0/1670381200 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f5fe8104000 con 0x7f5fcc06ec00 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (5m) 46s ago 16m 13.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (2m) 46s ago 16m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (2m) 19s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (5m) 46s ago 15m 72.2M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (9m) 46s ago 9m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (8m) 46s ago 8m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (8m) 19s ago 8m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (8m) 19s ago 8m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (5m) 46s ago 17m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:28:43.670 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (4m) 19s ago 14m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (4m) 46s ago 17m 45.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (3m) 19s ago 14m 35.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (5m) 46s ago 16m 7931k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 19s ago 14m 7386k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (114s) 46s ago 13m 48.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (84s) 46s ago 13m 48.4M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (54s) 46s ago 12m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (23s) 19s ago 11m 11.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (11m) 19s ago 11m 47.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7133906317db 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (10m) 19s ago 10m 46.6M 2581M 17.2.7-847-g45317f95 b333581ccdf5 82e996072d71 2024-04-16T07:28:43.671 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (4m) 46s ago 15m 57.6M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 msgr2=0x7f5fcc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 0x7f5fcc0710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8007fd0 tx=0x7f5fd8009040 comp rx=0 tx=0).stop 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 msgr2=0x7f5fe8115240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400b970 tx=0x7f5fd4009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 shutdown_connections 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5fcc06ec00 0x7f5fcc0710b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fe8071e20 0x7f5fe810fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 --2- 172.21.15.26:0/1670381200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe8110220 0x7f5fe8115240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 >> 172.21.15.26:0/1670381200 conn(0x7f5fe80f8580 msgr2=0x7f5fe80f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:43.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 shutdown_connections 2024-04-16T07:28:43.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.673+0000 7f5feec00700 1 -- 172.21.15.26:0/1670381200 wait complete. 2024-04-16T07:28:43.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:43 smithi026 ceph-mon[92215]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 -- 172.21.15.26:0/703393289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0103420 msgr2=0x7f9ac0103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 --2- 172.21.15.26:0/703393289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0103420 0x7f9ac0103830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004940 tx=0x7f9ab4009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 -- 172.21.15.26:0/703393289 shutdown_connections 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 --2- 172.21.15.26:0/703393289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac01046d0 0x7f9ac0104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 --2- 172.21.15.26:0/703393289 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac0103420 0x7f9ac0103830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 -- 172.21.15.26:0/703393289 >> 172.21.15.26:0/703393289 conn(0x7f9ac0072640 msgr2=0x7f9ac0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 -- 172.21.15.26:0/703393289 shutdown_connections 2024-04-16T07:28:43.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.818+0000 7f9ac8600700 1 -- 172.21.15.26:0/703393289 wait complete. 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.819+0000 7f9ac8600700 1 Processor -- start 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.819+0000 7f9ac8600700 1 -- start start 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac8600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0103420 0x7f9ac0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac8600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac8600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0114aa0 con 0x7f9ac01046d0 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac8600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0114c10 con 0x7f9ac0103420 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47716/0 (socket says 172.21.15.26:47716) 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 -- 172.21.15.26:0/972832815 learned_addr learned my addr 172.21.15.26:0/972832815 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 -- 172.21.15.26:0/972832815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0103420 msgr2=0x7f9ac0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0103420 0x7f9ac0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.820+0000 7f9ac5200700 1 -- 172.21.15.26:0/972832815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4005040 con 0x7f9ac01046d0 2024-04-16T07:28:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9ac5200700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c830 tx=0x7f9aac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aac00a760 con 0x7f9ac01046d0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9aac00ad90 con 0x7f9ac01046d0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0119670 con 0x7f9ac01046d0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aac0113a0 con 0x7f9ac01046d0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.821+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0119c30 con 0x7f9ac01046d0 2024-04-16T07:28:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.822+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac004c710 con 0x7f9ac01046d0 2024-04-16T07:28:43.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.823+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9aac012400 con 0x7f9ac01046d0 2024-04-16T07:28:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.824+0000 7f9abd600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 0x7f9aa80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.825+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9aac08e890 con 0x7f9ac01046d0 2024-04-16T07:28:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.825+0000 7f9ac5c00700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 0x7f9aa80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.825+0000 7f9ac5c00700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 0x7f9aa80710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004f90 tx=0x7f9ab4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:43.829+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9aac05b1f0 con 0x7f9ac01046d0 2024-04-16T07:28:44.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:44.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:28:44.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 27) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.098+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 32 ==== client_session(renewcaps seq 27) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:28:44.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:44.098+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 27) v1 from mds.0 2024-04-16T07:28:44.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.151+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9ac0119f00 con 0x7f9ac01046d0 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.152+0000 7f9abd600700 1 -- 172.21.15.26:0/972832815 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f9aac05ad80 con 0x7f9ac01046d0 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:28:44.151 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 5, 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 8 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:28:44.152 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:28:44.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.155+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 msgr2=0x7f9aa80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.155+0000 7f9ac8600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 0x7f9aa80710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004f90 tx=0x7f9ab4009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.155+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 msgr2=0x7f9ac01144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.155+0000 7f9ac8600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c830 tx=0x7f9aac004470 comp rx=0 tx=0).stop 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 shutdown_connections 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9aa806ec00 0x7f9aa80710b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ac0103420 0x7f9ac0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 --2- 172.21.15.26:0/972832815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ac01046d0 0x7f9ac01144a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 >> 172.21.15.26:0/972832815 conn(0x7f9ac0072640 msgr2=0x7f9ac0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:44.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 shutdown_connections 2024-04-16T07:28:44.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.156+0000 7f9ac8600700 1 -- 172.21.15.26:0/972832815 wait complete. 2024-04-16T07:28:44.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 -- 172.21.15.26:0/1771985333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9381046d0 msgr2=0x7fb938104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 --2- 172.21.15.26:0/1771985333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9381046d0 0x7fb938104b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb924004940 tx=0x7fb924009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:44.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 -- 172.21.15.26:0/1771985333 shutdown_connections 2024-04-16T07:28:44.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 --2- 172.21.15.26:0/1771985333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9381046d0 0x7fb938104b40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 --2- 172.21.15.26:0/1771985333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb938103420 0x7fb938103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 -- 172.21.15.26:0/1771985333 >> 172.21.15.26:0/1771985333 conn(0x7fb938072640 msgr2=0x7fb938074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 -- 172.21.15.26:0/1771985333 shutdown_connections 2024-04-16T07:28:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.316+0000 7fb93d200700 1 -- 172.21.15.26:0/1771985333 wait complete. 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 Processor -- start 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 -- start start 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 0x7fb938114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938114a90 con 0x7fb938103420 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.317+0000 7fb93d200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938114c00 con 0x7fb9381046d0 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47742/0 (socket says 172.21.15.26:47742) 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 -- 172.21.15.26:0/2431506425 learned_addr learned my addr 172.21.15.26:0/2431506425 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935400700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 0x7fb938114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 -- 172.21.15.26:0/2431506425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 msgr2=0x7fb938114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 0x7fb938114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 -- 172.21.15.26:0/2431506425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb924005040 con 0x7fb938103420 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935400700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 0x7fb938114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.318+0000 7fb935e00700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c8c0 tx=0x7fb9280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.319+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92800a980 con 0x7fb938103420 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.319+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb92800ce60 con 0x7fb938103420 2024-04-16T07:28:44.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.319+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9280115e0 con 0x7fb938103420 2024-04-16T07:28:44.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.319+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb938119660 con 0x7fb938103420 2024-04-16T07:28:44.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.319+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938119c20 con 0x7fb938103420 2024-04-16T07:28:44.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.320+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93804c710 con 0x7fb938103420 2024-04-16T07:28:44.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.321+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb928012400 con 0x7fb938103420 2024-04-16T07:28:44.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.322+0000 7fb92e000700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 0x7fb91c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.322+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb92808e980 con 0x7fb938103420 2024-04-16T07:28:44.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.323+0000 7fb935400700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 0x7fb91c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.323+0000 7fb935400700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 0x7fb91c0710b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb92400b020 tx=0x7fb924009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:44.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.328+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb92805b2e0 con 0x7fb938103420 2024-04-16T07:28:44.332 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 podman[94901]: 2024-04-16 07:28:44.047782382 +0000 UTC m=+0.739472080 container remove 262a1b6c8155359e59b869cbb03242f7c5153690bc117e28f6ea8292be7c6af0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T07:28:44.332 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-16T07:28:44.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 systemd[1]: Stopped Ceph osd.4 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:28:44.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.4.service: Consumed 12.008s CPU time. 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:44.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:44.383+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:44.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.613+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb93810bf10 con 0x7fb938103420 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.615+0000 7fb92e000700 1 -- 172.21.15.26:0/2431506425 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fb9280126b0 con 0x7fb938103420 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:44.614 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:28:44.615 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:44.616 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 msgr2=0x7fb91c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 0x7fb91c0710b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb92400b020 tx=0x7fb924009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:44.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 msgr2=0x7fb938113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c8c0 tx=0x7fb9280044c0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 shutdown_connections 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb91c06ec00 0x7fb91c0710b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb938103420 0x7fb938113f50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 --2- 172.21.15.26:0/2431506425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9381046d0 0x7fb938114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 >> 172.21.15.26:0/2431506425 conn(0x7fb938072640 msgr2=0x7fb938073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 shutdown_connections 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.618+0000 7fb93d200700 1 -- 172.21.15.26:0/2431506425 wait complete. 2024-04-16T07:28:44.618 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:28:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:44 smithi026 ceph-mon[92215]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:44 smithi026 ceph-mon[92215]: pgmap v142: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:44 smithi026 ceph-mon[92215]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:44 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/972832815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 -- 172.21.15.26:0/3559155271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fe050 msgr2=0x7f23a40fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 --2- 172.21.15.26:0/3559155271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fe050 0x7f23a40fe4c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2390004940 tx=0x7f2390009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 -- 172.21.15.26:0/3559155271 shutdown_connections 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 --2- 172.21.15.26:0/3559155271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fe050 0x7f23a40fe4c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 --2- 172.21.15.26:0/3559155271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fce50 0x7f23a40fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 -- 172.21.15.26:0/3559155271 >> 172.21.15.26:0/3559155271 conn(0x7f23a40f8580 msgr2=0x7f23a40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 -- 172.21.15.26:0/3559155271 shutdown_connections 2024-04-16T07:28:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.782+0000 7f23aac00700 1 -- 172.21.15.26:0/3559155271 wait complete. 2024-04-16T07:28:44.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 Processor -- start 2024-04-16T07:28:44.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 -- start start 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 0x7f23a41101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41107a0 con 0x7f23a40fce50 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.783+0000 7f23aac00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4110ff0 con 0x7f23a40fe050 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47770/0 (socket says 172.21.15.26:47770) 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 -- 172.21.15.26:0/566258186 learned_addr learned my addr 172.21.15.26:0/566258186 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3400700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 0x7f23a41101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 -- 172.21.15.26:0/566258186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 msgr2=0x7f23a41101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 0x7f23a41101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 -- 172.21.15.26:0/566258186 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390005040 con 0x7f23a40fce50 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3400700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 0x7f23a41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.784+0000 7f23a3e00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2394002a80 tx=0x7f2394004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.785+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2394007cc0 con 0x7f23a40fce50 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.785+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f239400f040 con 0x7f23a40fce50 2024-04-16T07:28:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.785+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a41112d0 con 0x7f23a40fce50 2024-04-16T07:28:44.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.785+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f239400dad0 con 0x7f23a40fce50 2024-04-16T07:28:44.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.785+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4111810 con 0x7f23a40fce50 2024-04-16T07:28:44.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.786+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a41019e0 con 0x7f23a40fce50 2024-04-16T07:28:44.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.787+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f239400dc30 con 0x7f23a40fce50 2024-04-16T07:28:44.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.788+0000 7f239be00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 0x7f23880710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:44.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.788+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f239408e6b0 con 0x7f23a40fce50 2024-04-16T07:28:44.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.788+0000 7f23a3400700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 0x7f23880710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:44.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.789+0000 7f23a3400700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 0x7f23880710c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239000b7a0 tx=0x7f2390009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:44.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:44.793+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f239405af10 con 0x7f23a40fce50 2024-04-16T07:28:44.819 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 systemd[1]: Starting Ceph osd.4 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:28:44.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:44 smithi104 ceph-mon[81027]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:44.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:44 smithi104 ceph-mon[81027]: pgmap v142: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:28:44.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:44 smithi104 ceph-mon[81027]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:44.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:44 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/972832815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:28:45.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.053+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23a4111a30 con 0x7f238806ec10 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.056+0000 7f239be00700 1 -- 172.21.15.26:0/566258186 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f23a4111a30 con 0x7f238806ec10 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:28:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "10/21 daemons upgraded", 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:28:45.056 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:28:45.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 msgr2=0x7f23880710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 0x7f23880710c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239000b7a0 tx=0x7f2390009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:45.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 msgr2=0x7f23a410fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2394002a80 tx=0x7f2394004600 comp rx=0 tx=0).stop 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 shutdown_connections 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f238806ec10 0x7f23880710c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23a40fce50 0x7f23a410fc60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 --2- 172.21.15.26:0/566258186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a40fe050 0x7f23a41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 >> 172.21.15.26:0/566258186 conn(0x7f23a40f8580 msgr2=0x7f23a40f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 shutdown_connections 2024-04-16T07:28:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.059+0000 7f23aac00700 1 -- 172.21.15.26:0/566258186 wait complete. 2024-04-16T07:28:45.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 podman[94998]: 2024-04-16 07:28:44.726492549 +0000 UTC m=+0.036539459 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:45.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:44 smithi104 podman[94998]: 2024-04-16 07:28:44.865556889 +0000 UTC m=+0.175603799 container create 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:28:45.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:45.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:45.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:45.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:45.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:45.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:45.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:45.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 -- 172.21.15.26:0/4149483750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff058071f50 msgr2=0x7ff058070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 --2- 172.21.15.26:0/4149483750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff058071f50 0x7ff058070460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff048004940 tx=0x7ff048009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 -- 172.21.15.26:0/4149483750 shutdown_connections 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 --2- 172.21.15.26:0/4149483750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0580709a0 0x7ff058070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 --2- 172.21.15.26:0/4149483750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff058071f50 0x7ff058070460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 -- 172.21.15.26:0/4149483750 >> 172.21.15.26:0/4149483750 conn(0x7ff0580f8530 msgr2=0x7ff0580fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 -- 172.21.15.26:0/4149483750 shutdown_connections 2024-04-16T07:28:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.222+0000 7ff05f000700 1 -- 172.21.15.26:0/4149483750 wait complete. 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 Processor -- start 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 -- start start 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff058071f50 0x7ff058110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058110750 con 0x7ff0580709a0 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.223+0000 7ff05f000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058110fa0 con 0x7ff058071f50 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47800/0 (socket says 172.21.15.26:47800) 2024-04-16T07:28:45.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 -- 172.21.15.26:0/3377062890 learned_addr learned my addr 172.21.15.26:0/3377062890 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057400700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff058071f50 0x7ff058110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 -- 172.21.15.26:0/3377062890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff058071f50 msgr2=0x7ff058110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff058071f50 0x7ff058110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 -- 172.21.15.26:0/3377062890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048005040 con 0x7ff0580709a0 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.224+0000 7ff057e00700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff048000c00 tx=0x7ff048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.225+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04800e030 con 0x7ff0580709a0 2024-04-16T07:28:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.225+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff04801c400 con 0x7ff0580709a0 2024-04-16T07:28:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.225+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04800b650 con 0x7ff0580709a0 2024-04-16T07:28:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.225+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff058111220 con 0x7ff0580709a0 2024-04-16T07:28:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.225+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff058111760 con 0x7ff0580709a0 2024-04-16T07:28:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.226+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05805f5f0 con 0x7ff0580709a0 2024-04-16T07:28:45.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.227+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff04801ca90 con 0x7ff0580709a0 2024-04-16T07:28:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.228+0000 7ff04fe00700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 0x7ff03c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:28:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.228+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff0480901d0 con 0x7ff0580709a0 2024-04-16T07:28:45.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.229+0000 7ff057400700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 0x7ff03c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:28:45.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.229+0000 7ff057400700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 0x7ff03c0710b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff044004600 tx=0x7ff04400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:28:45.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.233+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff04805cb30 con 0x7ff0580709a0 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 200 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:45.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.383+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:45.508 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:45 smithi104 podman[94998]: 2024-04-16 07:28:45.128826723 +0000 UTC m=+0.438873686 container init 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T07:28:45.508 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:45 smithi104 podman[94998]: 2024-04-16 07:28:45.14084078 +0000 UTC m=+0.450887684 container start 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:28:45.508 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:45 smithi104 podman[94998]: 2024-04-16 07:28:45.192612689 +0000 UTC m=+0.502659605 container attach 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:28:45.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.552+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff058111a90 con 0x7ff0580709a0 2024-04-16T07:28:45.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.553+0000 7ff04fe00700 1 -- 172.21.15.26:0/3377062890 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+544 (secure 0 0 0) 0x7ff04805c6c0 con 0x7ff0580709a0 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 7/72 objects degraded (9.722%), 5 pgs degraded 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: osd.4 (root=default,host=smithi104) is down 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 7/72 objects degraded (9.722%), 5 pgs degraded 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.6 is active+undersized+degraded, acting [1,3] 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.f is active+undersized+degraded, acting [0,5] 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.13 is active+undersized+degraded, acting [0,2] 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-16T07:28:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.1f is active+undersized+degraded, acting [0,3] 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 msgr2=0x7ff03c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 0x7ff03c0710b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff044004600 tx=0x7ff04400b040 comp rx=0 tx=0).stop 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 msgr2=0x7ff05810fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff048000c00 tx=0x7ff048009e30 comp rx=0 tx=0).stop 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 shutdown_connections 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff03c06ec00 0x7ff03c0710b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0580709a0 0x7ff05810fc10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 --2- 172.21.15.26:0/3377062890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff058071f50 0x7ff058110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 >> 172.21.15.26:0/3377062890 conn(0x7ff0580f8530 msgr2=0x7ff0580f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 shutdown_connections 2024-04-16T07:28:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:28:45.556+0000 7ff05f000700 1 -- 172.21.15.26:0/3377062890 wait complete. 2024-04-16T07:28:45.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:45 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2431506425' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:28:45.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:45 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2431506425' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:28:45.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 95087 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 406, success, outsize: 120 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.880+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.880+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.880+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.880+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:45.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:45.880+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:46.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:46.097+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 bash[94998]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T07:28:46.334 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T07:28:46.335 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T07:28:46.335 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T07:28:46.335 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T07:28:46.335 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate[95012]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T07:28:46.335 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 podman[94998]: 2024-04-16 07:28:46.073329616 +0000 UTC m=+1.383376516 container died 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-16T07:28:46.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:46.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:46.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:46.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:46.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:46.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:46.384+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:46 smithi026 ceph-mon[92215]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:46 smithi026 ceph-mon[92215]: pgmap v143: 65 pgs: 5 active+undersized+degraded, 11 peering, 18 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:46 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:46 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3377062890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:28:46.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-mon[81027]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:28:46.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-mon[81027]: pgmap v143: 65 pgs: 5 active+undersized+degraded, 11 peering, 18 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T07:28:46.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:46.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:46 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3377062890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:28:47.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:47.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:47.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:47.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:47.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:47.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:47.098+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:47.193 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:46 smithi104 podman[95131]: 2024-04-16 07:28:46.837301224 +0000 UTC m=+0.742873778 container remove 4a21f8b99ecb16f2df10bf0c15621ef06008608410f591492106cef2757901d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-activate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:47.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:47.384+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:47.499 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 podman[95190]: 2024-04-16 07:28:47.102731722 +0000 UTC m=+0.039143255 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:28:47.499 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 podman[95190]: 2024-04-16 07:28:47.229912043 +0000 UTC m=+0.166323607 container create 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:28:47.499 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 podman[95190]: 2024-04-16 07:28:47.500275515 +0000 UTC m=+0.436687106 container init 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:28:47.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 podman[95190]: 2024-04-16 07:28:47.510815587 +0000 UTC m=+0.447227122 container start 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:28:47.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 ceph-osd[95208]: -- 172.21.15.104:0/2423425817 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564956215680 con 0x5649561dec00 2024-04-16T07:28:47.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 bash[95190]: 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 2024-04-16T07:28:47.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:47 smithi104 systemd[1]: Started Ceph osd.4 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:28:48.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:48.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:48.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:48.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:48.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:48.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:48.099+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:48.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:48.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:48.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:48.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:48.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:48.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:48.384+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:48.582 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:48 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:28:48.107+0000 7f87afc887c0 -1 Falling back to public interface 2024-04-16T07:28:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:48 smithi104 ceph-mon[81027]: pgmap v144: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:48 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:49.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:49.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:49.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:49.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:49.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:49.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:49.098+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:48 smithi026 ceph-mon[92215]: pgmap v144: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:48 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:28:49.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:49.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:49.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:49.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:49.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:49.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:49.384+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:50.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:50.099+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.384+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.384+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.384+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.385+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:50.385+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:50 smithi026 ceph-mon[92215]: pgmap v145: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:50.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:50 smithi104 ceph-mon[81027]: pgmap v145: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:28:51.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:51.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:51.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:51.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:51.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:51.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:51.099+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:51.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:51.385+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:52.082 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:51 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:28:51.813+0000 7f87afc887c0 -1 osd.4 70 log_to_monitors true 2024-04-16T07:28:52.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:52.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:52.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:52.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:52.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:52.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:52.100+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 27) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:52.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:28:52.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:52.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.385+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:52.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.386+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 31 ==== client_session(renewcaps seq 27) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:52.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:52.386+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 27) v1 from mds.0 2024-04-16T07:28:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:52 smithi026 ceph-mon[92215]: pgmap v146: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:52 smithi026 ceph-mon[92215]: from='osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:28:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:52 smithi026 ceph-mon[92215]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:28:53.082 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:28:52.774+0000 7f879f000700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:28:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-mon[81027]: pgmap v146: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 719 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-mon[81027]: from='osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:28:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:52 smithi104 ceph-mon[81027]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:53.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:53.100+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:53.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:53.385+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:53 smithi104 ceph-mon[81027]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T07:28:54.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:53 smithi104 ceph-mon[81027]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T07:28:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:53 smithi104 ceph-mon[81027]: from='osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:53 smithi104 ceph-mon[81027]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:54.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:54.100+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:53 smithi026 ceph-mon[92215]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T07:28:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:53 smithi026 ceph-mon[92215]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T07:28:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:53 smithi026 ceph-mon[92215]: from='osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:53 smithi026 ceph-mon[92215]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:28:54.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:54.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:54.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:54.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:54.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:54.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:54.385+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: pgmap v148: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:28:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815] boot 2024-04-16T07:28:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: osdmap e74: 6 total, 6 up, 6 in 2024-04-16T07:28:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:28:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:54 smithi104 ceph-mon[81027]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T07:28:55.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:55.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:55.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:55.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:55.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:55.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:55.100+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: pgmap v148: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: osd.4 [v2:172.21.15.104:6808/6937815,v1:172.21.15.104:6809/6937815] boot 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: osdmap e74: 6 total, 6 up, 6 in 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T07:28:55.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:54 smithi026 ceph-mon[92215]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.385+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.385+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.386+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.386+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:55.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:55.386+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:56.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:56.101+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:56.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:56.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:56.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:56.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:56.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:56.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:56.386+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:57.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:56 smithi104 ceph-mon[81027]: pgmap v151: 65 pgs: 14 peering, 5 active+undersized+degraded, 11 active+undersized, 35 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:57.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:57.101+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:56 smithi026 ceph-mon[92215]: pgmap v151: 65 pgs: 14 peering, 5 active+undersized+degraded, 11 active+undersized, 35 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 201 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:57.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.386+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:57.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 96908 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.790+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 408, success, outsize: 120 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.790+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.790+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.790+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.790+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 96986 2024-04-16T07:28:57.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 410, success, outsize: 120 2024-04-16T07:28:57.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.806+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.806+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.806+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.806+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.806+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97067 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 412, success, outsize: 120 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.828+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.828+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.828+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.828+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.828+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97154 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 414, success, outsize: 120 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.849+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.849+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.849+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.849+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.849+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97238 2024-04-16T07:28:57.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 416, success, outsize: 120 2024-04-16T07:28:57.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.872+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.872+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.872+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.872+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.872+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97316 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 418, success, outsize: 120 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.892+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.892+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.892+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.892+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.892+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97397 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 420, success, outsize: 120 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.914+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.914+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.914+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.914+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.914+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97484 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 422, success, outsize: 120 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.938+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.938+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.938+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.938+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.938+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97568 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 424, success, outsize: 120 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.961+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.961+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.961+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.961+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.961+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:57.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97646 2024-04-16T07:28:57.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 426, success, outsize: 120 2024-04-16T07:28:57.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.983+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:57.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.983+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:57.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.983+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:57.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.983+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:57.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:57.983+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97727 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 428, success, outsize: 120 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.002+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.002+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.002+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.002+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.002+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97814 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 430, success, outsize: 120 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.021+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.021+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.021+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.021+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.021+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97898 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 432, success, outsize: 120 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.040+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.040+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.040+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.040+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.040+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97976 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 434, success, outsize: 120 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.059+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.059+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.059+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.059+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.059+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98057 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 436, success, outsize: 120 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.080+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.080+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.080+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.080+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.080+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:58.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:58.101+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98144 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 438, success, outsize: 120 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.102+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.102+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.102+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.102+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.102+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98228 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 440, success, outsize: 120 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.124+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.124+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.124+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.124+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.124+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98306 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 442, success, outsize: 120 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.141+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.141+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.141+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.141+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.141+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98387 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 444, success, outsize: 120 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.160+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.160+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.160+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.160+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.160+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98474 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 446, success, outsize: 120 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.180+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.180+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.180+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.180+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.180+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98558 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 448, success, outsize: 120 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.198+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.198+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.198+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.198+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.198+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98636 2024-04-16T07:28:58.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 450, success, outsize: 120 2024-04-16T07:28:58.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.216+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.216+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.216+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.216+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.216+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98717 2024-04-16T07:28:58.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 452, success, outsize: 120 2024-04-16T07:28:58.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.233+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.233+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.233+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.233+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.233+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98804 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 454, success, outsize: 120 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.252+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.252+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.252+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.252+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.252+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98888 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 456, success, outsize: 120 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.270+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.270+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.270+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.270+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.270+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98966 2024-04-16T07:28:58.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 458, success, outsize: 120 2024-04-16T07:28:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.285+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.285+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.285+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.285+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.285+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99047 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 460, success, outsize: 120 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.301+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.301+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.301+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.301+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.301+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99134 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 462, success, outsize: 120 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.317+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.317+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.317+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.317+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:28:58.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.317+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:28:58.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:58.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:58.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:58.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:58.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:58.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:58.386+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:59.027 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:28:58 smithi104 ceph-mon[81027]: pgmap v152: 65 pgs: 14 peering, 2 active+undersized+degraded, 4 active+undersized, 45 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T07:28:59.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:28:59.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:28:59.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:28:59.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:28:59.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:28:59.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:28:59.102+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:28:59.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:28:58 smithi026 ceph-mon[92215]: pgmap v152: 65 pgs: 14 peering, 2 active+undersized+degraded, 4 active+undersized, 45 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T07:28:59.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:28:59.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:28:59.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:28:59.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:28:59.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:28:59.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:28:59.386+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:00.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:00.102+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:00.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.386+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.387+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.387+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:00.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:00.387+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: Health check update: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: pgmap v153: 65 pgs: 14 peering, 51 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.469 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T07:29:00.470 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T07:29:00.470 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:00 smithi104 ceph-mon[81027]: Upgrade: osd.5 is safe to restart 2024-04-16T07:29:00.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:00.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: Health check update: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: pgmap v153: 65 pgs: 14 peering, 51 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T07:29:00.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:00 smithi026 ceph-mon[92215]: Upgrade: osd.5 is safe to restart 2024-04-16T07:29:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.101+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.101+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.101+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.102+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:01.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.102+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:01.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:01.102+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:01.387+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:01.405 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:01 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded) 2024-04-16T07:29:01.405 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:01 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:29:01.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:01 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded) 2024-04-16T07:29:01.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:01 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:29:02.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:02.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:02.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:02.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:02.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:02.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:02.103+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:02.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:02.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:02.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:02.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:02.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:02.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:02.387+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: pgmap v154: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: Upgrade: Updating osd.5 2024-04-16T07:29:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:29:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:02 smithi104 ceph-mon[81027]: Deploying daemon osd.5 on smithi104 2024-04-16T07:29:03.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:03.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:03.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:03.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:03.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:03.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:03.102+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: pgmap v154: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: Upgrade: Updating osd.5 2024-04-16T07:29:03.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:03.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T07:29:03.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:03.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:02 smithi026 ceph-mon[92215]: Deploying daemon osd.5 on smithi104 2024-04-16T07:29:03.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:03.387+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 28) v5 -- 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:29:04.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 33 ==== client_session(renewcaps seq 28) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:29:04.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:04.103+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 28) v1 from mds.0 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:04.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:04.388+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:04.950 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:04 smithi104 ceph-mon[81027]: pgmap v155: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:04.951 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:04.951 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:05.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:05.103+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:04 smithi026 ceph-mon[92215]: pgmap v155: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:05.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:05.388+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:06.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:06.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:06.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:06.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:06.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:06.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:06.104+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.388+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.388+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.388+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.389+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:06.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:06.389+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:06.832 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:06 smithi104 systemd[1]: Stopping Ceph osd.5 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:07.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:07.103+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:06 smithi026 ceph-mon[92215]: pgmap v156: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:07.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:06 smithi104 ceph-mon[81027]: pgmap v156: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:07.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:06 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[54650]: 2024-04-16T07:29:06.954+0000 7f0cb1a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:29:07.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:06 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[54650]: 2024-04-16T07:29:06.954+0000 7f0cb1a00700 -1 osd.5 75 *** Got signal Terminated *** 2024-04-16T07:29:07.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:06 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[54650]: 2024-04-16T07:29:06.954+0000 7f0cb1a00700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:07.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:07.389+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:08.104+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:08.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:07 smithi026 ceph-mon[92215]: osd.5 marked itself down and dead 2024-04-16T07:29:08.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:07 smithi104 ceph-mon[81027]: osd.5 marked itself down and dead 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:08.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:08.389+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:08.622 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:08 smithi104 podman[99598]: 2024-04-16 07:29:08.3645145 +0000 UTC m=+1.579318085 container died 82e996072d7132fde03105608f76fd7d998bd0430368d0e32482083f1c9cd0d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-16T07:29:08.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:08 smithi104 ceph-mon[81027]: pgmap v157: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:08.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:08 smithi104 ceph-mon[81027]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:29:08.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:08 smithi104 ceph-mon[81027]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T07:29:08.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:08 smithi104 ceph-mon[81027]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T07:29:08.948 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:08 smithi104 podman[99598]: 2024-04-16 07:29:08.738771448 +0000 UTC m=+1.953575047 container cleanup 82e996072d7132fde03105608f76fd7d998bd0430368d0e32482083f1c9cd0d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:29:08.948 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:08 smithi104 bash[99598]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:09.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:09.104+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:08 smithi026 ceph-mon[92215]: pgmap v157: 65 pgs: 65 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:08 smithi026 ceph-mon[92215]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T07:29:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:08 smithi026 ceph-mon[92215]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T07:29:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:08 smithi026 ceph-mon[92215]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T07:29:09.332 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:08 smithi104 podman[99617]: 2024-04-16 07:29:08.948243663 +0000 UTC m=+0.573465490 container remove 82e996072d7132fde03105608f76fd7d998bd0430368d0e32482083f1c9cd0d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T07:29:09.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:09.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:09.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:09.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:09.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:09.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:09.389+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:09 smithi104 ceph-mon[81027]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T07:29:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:10.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:09 smithi104 podman[99706]: 2024-04-16 07:29:09.708477636 +0000 UTC m=+0.034015587 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:29:10.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:09 smithi104 podman[99706]: 2024-04-16 07:29:09.824218286 +0000 UTC m=+0.149756272 container create 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T07:29:10.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:10.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:10.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:10.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:10.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:10.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:10.104+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:09 smithi026 ceph-mon[92215]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T07:29:10.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:09 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:10.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:10.390+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:10.582 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:10 smithi104 podman[99706]: 2024-04-16 07:29:10.136450625 +0000 UTC m=+0.461988618 container init 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-16T07:29:10.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:10 smithi104 podman[99706]: 2024-04-16 07:29:10.149457238 +0000 UTC m=+0.474995201 container start 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T07:29:10.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:10 smithi104 podman[99706]: 2024-04-16 07:29:10.234830151 +0000 UTC m=+0.560368193 container attach 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-16T07:29:10.895 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:10 smithi104 podman[99706]: 2024-04-16 07:29:10.604472431 +0000 UTC m=+0.930010381 container died 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-16T07:29:11.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:11.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:11.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:11.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:11.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:11.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:11.104+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:11.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:10 smithi104 ceph-mon[81027]: pgmap v160: 65 pgs: 13 stale+active+clean, 52 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:11.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:10 smithi026 ceph-mon[92215]: pgmap v160: 65 pgs: 13 stale+active+clean, 52 active+clean; 584 KiB data, 455 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:11.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:11.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:11.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:11.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:11.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:11.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:11.390+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:11.582 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 podman[99735]: 2024-04-16 07:29:11.221357163 +0000 UTC m=+0.601780257 container remove 518cd8bee1ca3eb5b480353e9775a6de3359a3fc7df55e632445884c71cd168d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-16T07:29:11.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-16T07:29:11.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 systemd[1]: Stopped Ceph osd.5 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:29:11.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 systemd[1]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.5.service: Consumed 12.285s CPU time. 2024-04-16T07:29:11.950 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 systemd[1]: Starting Ceph osd.5 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.104+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.104+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.104+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.104+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.105+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:12.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:12.105+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:12.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:11 smithi026 ceph-mon[92215]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:12.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:11 smithi104 ceph-mon[81027]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:12.291 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:11 smithi104 podman[99844]: 2024-04-16 07:29:11.873493881 +0000 UTC m=+0.032598639 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:29:12.291 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:12 smithi104 podman[99844]: 2024-04-16 07:29:12.01414319 +0000 UTC m=+0.173247951 container create 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-16T07:29:12.291 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:12 smithi104 podman[99844]: 2024-04-16 07:29:12.292522238 +0000 UTC m=+0.451627053 container init 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T07:29:12.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:12.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 28) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 229 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:12.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.390+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:12.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.391+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 32 ==== client_session(renewcaps seq 28) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:12.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:12.391+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 28) v1 from mds.0 2024-04-16T07:29:12.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:12 smithi104 podman[99844]: 2024-04-16 07:29:12.304355535 +0000 UTC m=+0.463460292 container start 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-16T07:29:12.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:12 smithi104 podman[99844]: 2024-04-16 07:29:12.39120567 +0000 UTC m=+0.550310422 container attach 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-16T07:29:13.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99934 2024-04-16T07:29:13.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.051+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:13.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 464, success, outsize: 120 2024-04-16T07:29:13.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.051+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:13.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.051+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:13.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.051+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:13.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.051+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:13.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:13.105+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:13.240 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:12 smithi104 ceph-mon[81027]: pgmap v161: 65 pgs: 3 active+undersized+degraded, 4 stale+active+clean, 18 active+undersized, 40 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T07:29:13.241 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 bash[99844]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T07:29:13.242 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate[99859]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T07:29:13.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:12 smithi026 ceph-mon[92215]: pgmap v161: 65 pgs: 3 active+undersized+degraded, 4 stale+active+clean, 18 active+undersized, 40 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T07:29:13.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:13.391+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:13.582 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:13 smithi104 podman[99844]: 2024-04-16 07:29:13.241757151 +0000 UTC m=+1.400861922 container died 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-16T07:29:14.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:14.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:14.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:14.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:14.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:14.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:14.105+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:14.332 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 podman[99978]: 2024-04-16 07:29:14.061335373 +0000 UTC m=+0.800613539 container remove 2968ccec709679ffcdf2ec97b5f2500faf0f7bbba64082978ec0b06e27550c8d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:14.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:14.391+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:14.689 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 podman[100040]: 2024-04-16 07:29:14.321183153 +0000 UTC m=+0.036793832 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:29:14.689 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 podman[100040]: 2024-04-16 07:29:14.428722936 +0000 UTC m=+0.144333649 container create ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T07:29:14.690 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 podman[100040]: 2024-04-16 07:29:14.690353928 +0000 UTC m=+0.405964661 container init ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T07:29:14.958 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 podman[100040]: 2024-04-16 07:29:14.700083371 +0000 UTC m=+0.415694120 container start ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:29:14.958 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 ceph-osd[100058]: -- 172.21.15.104:0/231939676 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b3dbcbd680 con 0x55b3dbc86c00 2024-04-16T07:29:14.959 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 bash[100040]: ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 2024-04-16T07:29:14.959 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:14 smithi104 systemd[1]: Started Ceph osd.5 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968. 2024-04-16T07:29:15.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:15.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:15.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:15.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:15.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:15.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:15.105+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:15.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:14 smithi026 ceph-mon[92215]: pgmap v162: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:15.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:14 smithi104 ceph-mon[81027]: pgmap v162: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:15.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:15.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:15.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:15.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:15.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:15.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:15.391+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:15.582 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:15 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:29:15.266+0000 7fb5a73ce7c0 -1 Falling back to public interface 2024-04-16T07:29:15.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2039496296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 msgr2=0x7fcc780ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2039496296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc780ffab0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc68004880 tx=0x7fcc68009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2039496296 shutdown_connections 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2039496296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 0x7fcc781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2039496296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc780ffab0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2039496296 >> 172.21.15.26:0/2039496296 conn(0x7fcc780f74c0 msgr2=0x7fcc780f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2039496296 shutdown_connections 2024-04-16T07:29:15.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.725+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2039496296 wait complete. 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.726+0000 7fcc7ca00700 1 Processor -- start 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.726+0000 7fcc7ca00700 1 -- start start 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc7ca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc7ca00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 0x7fcc78110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc7ca00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78110860 con 0x7fcc780ffff0 2024-04-16T07:29:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc7ca00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78110fa0 con 0x7fcc78066270 2024-04-16T07:29:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc75400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 0x7fcc78110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.26:46406/0 (socket says 172.21.15.26:46406) 2024-04-16T07:29:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.727+0000 7fcc75e00700 1 -- 172.21.15.26:0/2836879427 learned_addr learned my addr 172.21.15.26:0/2836879427 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc75e00700 1 -- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 msgr2=0x7fcc78110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc75e00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 0x7fcc78110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc75e00700 1 -- 172.21.15.26:0/2836879427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68005040 con 0x7fcc78066270 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc75e00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc680040d0 tx=0x7fcc68009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6800e030 con 0x7fcc78066270 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc68013e60 con 0x7fcc78066270 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc78111220 con 0x7fcc78066270 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.728+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc680123c0 con 0x7fcc78066270 2024-04-16T07:29:15.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.729+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc781117e0 con 0x7fcc78066270 2024-04-16T07:29:15.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.730+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7805f5f0 con 0x7fcc78066270 2024-04-16T07:29:15.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.731+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc6801c400 con 0x7fcc78066270 2024-04-16T07:29:15.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.732+0000 7fcc6e000700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 0x7fcc5c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:15.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.732+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc68090250 con 0x7fcc78066270 2024-04-16T07:29:15.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.732+0000 7fcc75400700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 0x7fcc5c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:15.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.733+0000 7fcc75400700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 0x7fcc5c0710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009b80 tx=0x7fcc6400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:15.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:15.741+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcc6805cbf0 con 0x7fcc78066270 2024-04-16T07:29:16.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.041+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc78111b00 con 0x7fcc5c06ec00 2024-04-16T07:29:16.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.043+0000 7fcc6e000700 1 -- 172.21.15.26:0/2836879427 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fcc78111b00 con 0x7fcc5c06ec00 2024-04-16T07:29:16.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 msgr2=0x7fcc5c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 0x7fcc5c0710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009b80 tx=0x7fcc6400b040 comp rx=0 tx=0).stop 2024-04-16T07:29:16.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 msgr2=0x7fcc7810fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc680040d0 tx=0x7fcc68009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 shutdown_connections 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcc5c06ec00 0x7fcc5c0710b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78066270 0x7fcc7810fd20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 --2- 172.21.15.26:0/2836879427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc780ffff0 0x7fcc78110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 >> 172.21.15.26:0/2836879427 conn(0x7fcc780f74c0 msgr2=0x7fcc780fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 shutdown_connections 2024-04-16T07:29:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.046+0000 7fcc7ca00700 1 -- 172.21.15.26:0/2836879427 wait complete. 2024-04-16T07:29:16.060 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:16.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:16.105+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 -- 172.21.15.26:0/671101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 msgr2=0x7fa948102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 --2- 172.21.15.26:0/671101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa948102390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa934004940 tx=0x7fa934009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 -- 172.21.15.26:0/671101567 shutdown_connections 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 --2- 172.21.15.26:0/671101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa948102390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 --2- 172.21.15.26:0/671101567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 0x7fa9480ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 -- 172.21.15.26:0/671101567 >> 172.21.15.26:0/671101567 conn(0x7fa9480f74c0 msgr2=0x7fa9480f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 -- 172.21.15.26:0/671101567 shutdown_connections 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.191+0000 7fa94f200700 1 -- 172.21.15.26:0/671101567 wait complete. 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 Processor -- start 2024-04-16T07:29:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 -- start start 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 0x7fa94810fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948110870 con 0x7fa9480fff70 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa94f200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948111010 con 0x7fa9480662f0 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa947400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa947400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57778/0 (socket says 172.21.15.26:57778) 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.192+0000 7fa947400700 1 -- 172.21.15.26:0/2534357485 learned_addr learned my addr 172.21.15.26:0/2534357485 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.193+0000 7fa947e00700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 0x7fa94810fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.193+0000 7fa947400700 1 -- 172.21.15.26:0/2534357485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 msgr2=0x7fa94810fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.193+0000 7fa947400700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 0x7fa94810fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.193+0000 7fa947400700 1 -- 172.21.15.26:0/2534357485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa934005040 con 0x7fa9480fff70 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.193+0000 7fa947400700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa93400b3b0 tx=0x7fa934009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.194+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa93400e040 con 0x7fa9480fff70 2024-04-16T07:29:16.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.194+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa934004290 con 0x7fa9480fff70 2024-04-16T07:29:16.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.194+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa948111290 con 0x7fa9480fff70 2024-04-16T07:29:16.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.194+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa93401c630 con 0x7fa9480fff70 2024-04-16T07:29:16.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.194+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa948111850 con 0x7fa9480fff70 2024-04-16T07:29:16.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.195+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94805f5f0 con 0x7fa9480fff70 2024-04-16T07:29:16.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.196+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa93400bbf0 con 0x7fa9480fff70 2024-04-16T07:29:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.197+0000 7fa93fe00700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 0x7fa92c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.197+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa93408fd40 con 0x7fa9480fff70 2024-04-16T07:29:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.197+0000 7fa947e00700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 0x7fa92c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.198+0000 7fa947e00700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 0x7fa92c0710c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa938006fd0 tx=0x7fa938008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:16.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.203+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa93405c620 con 0x7fa9480fff70 2024-04-16T07:29:16.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:16.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:16.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:16.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:16.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:16.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:16.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:16.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:16.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:16.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:16.391+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:16.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.446+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa948111ba0 con 0x7fa92c06ec10 2024-04-16T07:29:16.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.448+0000 7fa93fe00700 1 -- 172.21.15.26:0/2534357485 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fa948111ba0 con 0x7fa92c06ec10 2024-04-16T07:29:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 msgr2=0x7fa92c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 0x7fa92c0710c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa938006fd0 tx=0x7fa938008040 comp rx=0 tx=0).stop 2024-04-16T07:29:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 msgr2=0x7fa9481101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa93400b3b0 tx=0x7fa934009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 shutdown_connections 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa92c06ec10 0x7fa92c0710c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9480662f0 0x7fa94810fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 --2- 172.21.15.26:0/2534357485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9480fff70 0x7fa9481101e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 >> 172.21.15.26:0/2534357485 conn(0x7fa9480f74c0 msgr2=0x7fa9480fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 shutdown_connections 2024-04-16T07:29:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.451+0000 7fa94f200700 1 -- 172.21.15.26:0/2534357485 wait complete. 2024-04-16T07:29:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 -- 172.21.15.26:0/4141493986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 msgr2=0x7fdc3c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 --2- 172.21.15.26:0/4141493986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c100720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc28004940 tx=0x7fdc28009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 -- 172.21.15.26:0/4141493986 shutdown_connections 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 --2- 172.21.15.26:0/4141493986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c100720 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 --2- 172.21.15.26:0/4141493986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.596+0000 7fdc43000700 1 -- 172.21.15.26:0/4141493986 >> 172.21.15.26:0/4141493986 conn(0x7fdc3c0fa950 msgr2=0x7fdc3c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.597+0000 7fdc43000700 1 -- 172.21.15.26:0/4141493986 shutdown_connections 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.597+0000 7fdc43000700 1 -- 172.21.15.26:0/4141493986 wait complete. 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.597+0000 7fdc43000700 1 Processor -- start 2024-04-16T07:29:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc43000700 1 -- start start 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc43000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c10fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc43000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc43000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c1107d0 con 0x7fdc3c1002b0 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc43000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c110fd0 con 0x7fdc3c0ff000 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc3b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc3b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57794/0 (socket says 172.21.15.26:57794) 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.598+0000 7fdc3b400700 1 -- 172.21.15.26:0/3421067223 learned_addr learned my addr 172.21.15.26:0/3421067223 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3be00700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c10fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3b400700 1 -- 172.21.15.26:0/3421067223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 msgr2=0x7fdc3c10fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3b400700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c10fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3b400700 1 -- 172.21.15.26:0/3421067223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28005040 con 0x7fdc3c1002b0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3be00700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc3b400700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009150 tx=0x7fdc28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc2800e040 con 0x7fdc3c1002b0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc2800bcc0 con 0x7fdc3c1002b0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc2801c630 con 0x7fdc3c1002b0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c111250 con 0x7fdc3c1002b0 2024-04-16T07:29:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.599+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c111790 con 0x7fdc3c1002b0 2024-04-16T07:29:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.600+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c05f5f0 con 0x7fdc3c1002b0 2024-04-16T07:29:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.602+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc28013420 con 0x7fdc3c1002b0 2024-04-16T07:29:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.602+0000 7fdc33e00700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 0x7fdc200710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.603+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdc2808f4f0 con 0x7fdc3c1002b0 2024-04-16T07:29:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.603+0000 7fdc3be00700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 0x7fdc200710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.604+0000 7fdc3be00700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 0x7fdc200710b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c007fd0 tx=0x7fdc2c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.607+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdc2805be50 con 0x7fdc3c1002b0 2024-04-16T07:29:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.848+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc3c111b80 con 0x7fdc2006ec00 2024-04-16T07:29:16.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.878+0000 7fdc33e00700 1 -- 172.21.15.26:0/3421067223 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fdc3c111b80 con 0x7fdc2006ec00 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (6m) 80s ago 16m 13.4M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (3m) 80s ago 16m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (2m) 25s ago 15m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (6m) 80s ago 16m 72.2M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (9m) 80s ago 9m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 e105c8dded60 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (9m) 80s ago 9m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:29:16.878 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (9m) 25s ago 9m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (9m) 25s ago 9m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (5m) 80s ago 18m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (4m) 25s ago 15m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (4m) 80s ago 18m 45.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (4m) 25s ago 15m 34.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (6m) 80s ago 16m 7931k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 25s ago 15m 7926k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (2m) 80s ago 14m 48.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (117s) 80s ago 13m 48.4M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (87s) 80s ago 13m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (56s) 25s ago 12m 50.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (29s) 25s ago 11m 11.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 starting - - - 2581M 2024-04-16T07:29:16.879 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (5m) 80s ago 15m 57.6M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 msgr2=0x7fdc200710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 0x7fdc200710b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c007fd0 tx=0x7fdc2c009040 comp rx=0 tx=0).stop 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 msgr2=0x7fdc3c1101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009150 tx=0x7fdc28009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 shutdown_connections 2024-04-16T07:29:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdc2006ec00 0x7fdc200710b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc3c0ff000 0x7fdc3c10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 --2- 172.21.15.26:0/3421067223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc3c1002b0 0x7fdc3c1101d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:16.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.881+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 >> 172.21.15.26:0/3421067223 conn(0x7fdc3c0fa950 msgr2=0x7fdc3c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:16.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.882+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 shutdown_connections 2024-04-16T07:29:16.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:16.882+0000 7fdc43000700 1 -- 172.21.15.26:0/3421067223 wait complete. 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 -- 172.21.15.26:0/2724964441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80662f0 msgr2=0x7fbcf80ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 --2- 172.21.15.26:0/2724964441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80662f0 0x7fbcf80ff800 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbcec004940 tx=0x7fbcec009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 -- 172.21.15.26:0/2724964441 shutdown_connections 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 --2- 172.21.15.26:0/2724964441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80ffd40 0x7fbcf8102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 --2- 172.21.15.26:0/2724964441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80662f0 0x7fbcf80ff800 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 -- 172.21.15.26:0/2724964441 >> 172.21.15.26:0/2724964441 conn(0x7fbcf80f7270 msgr2=0x7fbcf80f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 -- 172.21.15.26:0/2724964441 shutdown_connections 2024-04-16T07:29:17.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.044+0000 7fbcff200700 1 -- 172.21.15.26:0/2724964441 wait complete. 2024-04-16T07:29:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.045+0000 7fbcff200700 1 Processor -- start 2024-04-16T07:29:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.045+0000 7fbcff200700 1 -- start start 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcff200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 0x7fbcf810d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcff200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcff200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf810e3a0 con 0x7fbcf80ffd40 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcff200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf810eba0 con 0x7fbcf80662f0 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcfd400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcfd400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57806/0 (socket says 172.21.15.26:57806) 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcfd400700 1 -- 172.21.15.26:0/263230721 learned_addr learned my addr 172.21.15.26:0/263230721 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.046+0000 7fbcfde00700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 0x7fbcf810d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcfd400700 1 -- 172.21.15.26:0/263230721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 msgr2=0x7fbcf810d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcfd400700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 0x7fbcf810d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcfd400700 1 -- 172.21.15.26:0/263230721 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec005040 con 0x7fbcf80ffd40 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcfde00700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 0x7fbcf810d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcfd400700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbce800eb40 tx=0x7fbce800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce8007cc0 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbce800f040 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce800ca10 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf810ee20 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.047+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf810f360 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.049+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf805f5f0 con 0x7fbcf80ffd40 2024-04-16T07:29:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.050+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbce800cb70 con 0x7fbcf80ffd40 2024-04-16T07:29:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.050+0000 7fbcf2000700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 0x7fbce00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.051+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbce808e3f0 con 0x7fbcf80ffd40 2024-04-16T07:29:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.051+0000 7fbcfde00700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 0x7fbce00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.052+0000 7fbcfde00700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 0x7fbce00710c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbcec013870 tx=0x7fbcec009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:17.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.056+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbce805ad80 con 0x7fbcf80ffd40 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:17.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:17.106+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:17.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:17 smithi026 ceph-mon[92215]: pgmap v163: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:17.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:17 smithi026 ceph-mon[92215]: from='client.44229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:17.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:17 smithi104 ceph-mon[81027]: pgmap v163: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:17.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:17 smithi104 ceph-mon[81027]: from='client.44229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.385+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbcf810f6d0 con 0x7fbcf80ffd40 2024-04-16T07:29:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.386+0000 7fbcf2000700 1 -- 172.21.15.26:0/263230721 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+732 (secure 0 0 0) 0x7fbce805a910 con 0x7fbcf80ffd40 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 9 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:29:17.387 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:29:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 msgr2=0x7fbce00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 0x7fbce00710c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbcec013870 tx=0x7fbcec009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 msgr2=0x7fbcf810dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbce800eb40 tx=0x7fbce800b040 comp rx=0 tx=0).stop 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 shutdown_connections 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fbce006ec10 0x7fbce00710c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcf80662f0 0x7fbcf810d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 --2- 172.21.15.26:0/263230721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcf80ffd40 0x7fbcf810dda0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 >> 172.21.15.26:0/263230721 conn(0x7fbcf80f7270 msgr2=0x7fbcf80f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 shutdown_connections 2024-04-16T07:29:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.389+0000 7fbcff200700 1 -- 172.21.15.26:0/263230721 wait complete. 2024-04-16T07:29:17.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:17.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:17.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:17.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:17.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:17.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:17.391+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 -- 172.21.15.26:0/943483424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fe0e0 msgr2=0x7f6fe40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 --2- 172.21.15.26:0/943483424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe40fe550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0004940 tx=0x7f6fd0009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 -- 172.21.15.26:0/943483424 shutdown_connections 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 --2- 172.21.15.26:0/943483424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe40fe550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 --2- 172.21.15.26:0/943483424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fcef0 0x7f6fe40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 -- 172.21.15.26:0/943483424 >> 172.21.15.26:0/943483424 conn(0x7f6fe40f8530 msgr2=0x7f6fe40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 -- 172.21.15.26:0/943483424 shutdown_connections 2024-04-16T07:29:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.550+0000 7f6feae00700 1 -- 172.21.15.26:0/943483424 wait complete. 2024-04-16T07:29:17.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.551+0000 7f6feae00700 1 Processor -- start 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.551+0000 7f6feae00700 1 -- start start 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6feae00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6feae00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6feae00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40708a0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6feae00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40709e0 con 0x7f6fe40fe0e0 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57818/0 (socket says 172.21.15.26:57818) 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 -- 172.21.15.26:0/1619898363 learned_addr learned my addr 172.21.15.26:0/1619898363 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3400700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 -- 172.21.15.26:0/1619898363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fe0e0 msgr2=0x7f6fe4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe4070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.552+0000 7f6fe3e00700 1 -- 172.21.15.26:0/1619898363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0005040 con 0x7f6fe40fcef0 2024-04-16T07:29:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6fe3e00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c8b0 tx=0x7f6fd4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd400a760 con 0x7f6fe40fcef0 2024-04-16T07:29:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd400ad90 con 0x7f6fe40fcef0 2024-04-16T07:29:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd40113a0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4070cc0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.553+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe4115ac0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.555+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe405f5f0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.556+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fd4012400 con 0x7f6fe40fcef0 2024-04-16T07:29:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.557+0000 7f6fdbe00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 0x7f6fc80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.557+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6fd408e840 con 0x7f6fe40fcef0 2024-04-16T07:29:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.557+0000 7f6fe3400700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 0x7f6fc80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:17.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.558+0000 7f6fe3400700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 0x7f6fc80710c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40723d0 tx=0x7f6fd0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:17.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.562+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6fd4057670 con 0x7f6fe40fcef0 2024-04-16T07:29:17.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.839+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6fe404bed0 con 0x7f6fe40fcef0 2024-04-16T07:29:17.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.840+0000 7f6fdbe00700 1 -- 172.21.15.26:0/1619898363 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f6fd405ad30 con 0x7f6fe40fcef0 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:e16 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:29:17.841 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:29:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:29:17.843 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:14480} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/4271222482,v1:172.21.15.26:6827/4271222482] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.843+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 msgr2=0x7f6fc80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.843+0000 7f6feae00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 0x7f6fc80710c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40723d0 tx=0x7f6fd0009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.843+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 msgr2=0x7f6fe4071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.843+0000 7f6feae00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c8b0 tx=0x7f6fd4004470 comp rx=0 tx=0).stop 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 shutdown_connections 2024-04-16T07:29:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6fc806ec10 0x7f6fc80710c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fe40fcef0 0x7f6fe4071df0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 --2- 172.21.15.26:0/1619898363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fe40fe0e0 0x7f6fe4070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 >> 172.21.15.26:0/1619898363 conn(0x7f6fe40f8530 msgr2=0x7f6fe40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 shutdown_connections 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:17.844+0000 7f6feae00700 1 -- 172.21.15.26:0/1619898363 wait complete. 2024-04-16T07:29:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 -- 172.21.15.26:0/966959033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4066dc0 msgr2=0x7f35c4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 --2- 172.21.15.26:0/966959033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4066dc0 0x7f35c4101df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35b4004940 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 -- 172.21.15.26:0/966959033 shutdown_connections 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 --2- 172.21.15.26:0/966959033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4102330 0x7f35c4104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 --2- 172.21.15.26:0/966959033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4066dc0 0x7f35c4101df0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 -- 172.21.15.26:0/966959033 >> 172.21.15.26:0/966959033 conn(0x7f35c40f73e0 msgr2=0x7f35c40f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:18.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.001+0000 7f35c9200700 1 -- 172.21.15.26:0/966959033 shutdown_connections 2024-04-16T07:29:18.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.002+0000 7f35c9200700 1 -- 172.21.15.26:0/966959033 wait complete. 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.002+0000 7f35c9200700 1 Processor -- start 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c9200700 1 -- start start 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c9200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4066dc0 0x7f35c410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c9200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c9200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c41106d0 con 0x7f35c4102330 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c9200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4110ed0 con 0x7f35c4066dc0 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35bbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35844/0 (socket says 172.21.15.26:35844) 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35bbe00700 1 -- 172.21.15.26:0/2460438240 learned_addr learned my addr 172.21.15.26:0/2460438240 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:18.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.003+0000 7f35c1e00700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4066dc0 0x7f35c410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35bbe00700 1 -- 172.21.15.26:0/2460438240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4066dc0 msgr2=0x7f35c410fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35bbe00700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4066dc0 0x7f35c410fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35bbe00700 1 -- 172.21.15.26:0/2460438240 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4005040 con 0x7f35c4102330 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35bbe00700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00eb40 tx=0x7f35ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ac007cc0 con 0x7f35c4102330 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35ac00f040 con 0x7f35c4102330 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c4111150 con 0x7f35c4102330 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.004+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ac00ca10 con 0x7f35c4102330 2024-04-16T07:29:18.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.005+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c4111690 con 0x7f35c4102330 2024-04-16T07:29:18.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.006+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c40fb0a0 con 0x7f35c4102330 2024-04-16T07:29:18.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.007+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35ac00cc80 con 0x7f35c4102330 2024-04-16T07:29:18.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.008+0000 7f35ba000700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 0x7f35a80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.008+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f35ac08e470 con 0x7f35c4102330 2024-04-16T07:29:18.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.008+0000 7f35c1e00700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 0x7f35a80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.009+0000 7f35c1e00700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 0x7f35a80710b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35b4013870 tx=0x7f35b4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:18.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.013+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f35ac05ae00 con 0x7f35c4102330 2024-04-16T07:29:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:18.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:18.106+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:18 smithi026 ceph-mon[92215]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:18 smithi026 ceph-mon[92215]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:18 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/263230721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:18.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:18 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1619898363' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:29:18.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.269+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c4111a30 con 0x7f35a806ec00 2024-04-16T07:29:18.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.271+0000 7f35ba000700 1 -- 172.21.15.26:0/2460438240 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f35c4111a30 con 0x7f35a806ec00 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "11/21 daemons upgraded", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T07:29:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:29:18.272 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 msgr2=0x7f35a80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 0x7f35a80710b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35b4013870 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 msgr2=0x7f35c41100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00eb40 tx=0x7f35ac00b040 comp rx=0 tx=0).stop 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 shutdown_connections 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f35a806ec00 0x7f35a80710b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4066dc0 0x7f35c410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 --2- 172.21.15.26:0/2460438240 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4102330 0x7f35c41100d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.274+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 >> 172.21.15.26:0/2460438240 conn(0x7f35c40f73e0 msgr2=0x7f35c40f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.275+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 shutdown_connections 2024-04-16T07:29:18.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.275+0000 7f35c9200700 1 -- 172.21.15.26:0/2460438240 wait complete. 2024-04-16T07:29:18.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:18 smithi104 ceph-mon[81027]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:18.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:18 smithi104 ceph-mon[81027]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:18.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:18 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/263230721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:18.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:18 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1619898363' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:29:18.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.391+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.392+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.392+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:18.392+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 -- 172.21.15.26:0/3144080617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fdff0 msgr2=0x7f555c0fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 --2- 172.21.15.26:0/3144080617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fdff0 0x7f555c0fe460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5548004940 tx=0x7f5548009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 -- 172.21.15.26:0/3144080617 shutdown_connections 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 --2- 172.21.15.26:0/3144080617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fdff0 0x7f555c0fe460 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 --2- 172.21.15.26:0/3144080617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fce80 0x7f555c0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.433+0000 7f5560c00700 1 -- 172.21.15.26:0/3144080617 >> 172.21.15.26:0/3144080617 conn(0x7f555c0f8580 msgr2=0x7f555c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.434+0000 7f5560c00700 1 -- 172.21.15.26:0/3144080617 shutdown_connections 2024-04-16T07:29:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.434+0000 7f5560c00700 1 -- 172.21.15.26:0/3144080617 wait complete. 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 Processor -- start 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 -- start start 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fdff0 0x7f555c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c072260 con 0x7f555c0fce80 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.435+0000 7f5560c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c070850 con 0x7f555c0fdff0 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35860/0 (socket says 172.21.15.26:35860) 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 -- 172.21.15.26:0/3845859378 learned_addr learned my addr 172.21.15.26:0/3845859378 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559400700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fdff0 0x7f555c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 -- 172.21.15.26:0/3845859378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fdff0 msgr2=0x7f555c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fdff0 0x7f555c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 -- 172.21.15.26:0/3845859378 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c005040 con 0x7f555c0fce80 2024-04-16T07:29:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.436+0000 7f5559e00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f554c0096d0 tx=0x7f554c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.437+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554c00e840 con 0x7f555c0fce80 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.437+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f554c00f040 con 0x7f555c0fce80 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.437+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554c013dc0 con 0x7f555c0fce80 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.437+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5548005040 con 0x7f555c0fce80 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.437+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c070ed0 con 0x7f555c0fce80 2024-04-16T07:29:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.438+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c04c710 con 0x7f555c0fce80 2024-04-16T07:29:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.439+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f554c00e9a0 con 0x7f555c0fce80 2024-04-16T07:29:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.440+0000 7f5552000700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 0x7f55400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.440+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f554c08e5e0 con 0x7f555c0fce80 2024-04-16T07:29:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.440+0000 7f5559400700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 0x7f55400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.441+0000 7f5559400700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 0x7f55400710b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5548009fd0 tx=0x7f5548009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.445+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f554c05aec0 con 0x7f555c0fce80 2024-04-16T07:29:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.768+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f555c071140 con 0x7f555c0fce80 2024-04-16T07:29:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.768+0000 7f5552000700 1 -- 172.21.15.26:0/3845859378 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+783 (secure 0 0 0) 0x7f554c05aa50 con 0x7f555c0fce80 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: osd.5 (root=default,host=smithi104) is down 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-04-16T07:29:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-04-16T07:29:18.769 INFO:teuthology.orchestra.run.smithi026.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 msgr2=0x7f55400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 0x7f55400710b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5548009fd0 tx=0x7f5548009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 msgr2=0x7f555c071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f554c0096d0 tx=0x7f554c009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 shutdown_connections 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f554006ec00 0x7f55400710b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f555c0fce80 0x7f555c071c60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 --2- 172.21.15.26:0/3845859378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f555c0fdff0 0x7f555c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.771+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 >> 172.21.15.26:0/3845859378 conn(0x7f555c0f8580 msgr2=0x7f555c0f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.772+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 shutdown_connections 2024-04-16T07:29:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:18.772+0000 7f5560c00700 1 -- 172.21.15.26:0/3845859378 wait complete. 2024-04-16T07:29:19.036 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:18 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:29:18.932+0000 7fb5a73ce7c0 -1 osd.5 75 log_to_monitors true 2024-04-16T07:29:19.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:19.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:19.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:19.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:19.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:19.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:19.106+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:19.332 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:29:19 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:29:19.077+0000 7fb596800700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T07:29:19.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:19 smithi104 ceph-mon[81027]: pgmap v164: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:19.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:19 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3845859378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:29:19.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:19 smithi104 ceph-mon[81027]: from='osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:29:19.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:19 smithi104 ceph-mon[81027]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:19.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:19.392+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:19 smithi026 ceph-mon[92215]: pgmap v164: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 456 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:19 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3845859378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:29:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:19 smithi026 ceph-mon[92215]: from='osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:29:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:19 smithi026 ceph-mon[92215]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:20.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:20.106+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:20.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:20.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:20.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:20.392+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:21.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:21.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:21.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:21.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:21.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:21.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:21.107+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:21.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: pgmap v166: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:21.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:29:21.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483] boot 2024-04-16T07:29:21.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: osdmap e79: 6 total, 6 up, 6 in 2024-04-16T07:29:21.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:29:21.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:21 smithi104 ceph-mon[81027]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T07:29:21.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:21.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:21.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:21.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:21.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:21.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:21.392+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: pgmap v166: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: osd.5 [v2:172.21.15.104:6816/2796309483,v1:172.21.15.104:6817/2796309483] boot 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: osdmap e79: 6 total, 6 up, 6 in 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T07:29:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:21 smithi026 ceph-mon[92215]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T07:29:22.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:22.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:22.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:22.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:22.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:22.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:22.107+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:22.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:22.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:22.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:22.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:22.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:22.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:22.392+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:22 smithi026 ceph-mon[92215]: pgmap v169: 65 pgs: 7 peering, 2 active+undersized+degraded, 19 active+undersized, 37 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T07:29:22.785 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:22.785 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:22.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:22 smithi104 ceph-mon[81027]: pgmap v169: 65 pgs: 7 peering, 2 active+undersized+degraded, 19 active+undersized, 37 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T07:29:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:22.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:23.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.051+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c0042f0 con 0x7fbf70008390 2024-04-16T07:29:23.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.052+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 5 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c0042f0 con 0x7fbf70008390 2024-04-16T07:29:23.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:23.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:23.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:23.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:23.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:23.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:23.107+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:23.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:23.393+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:24.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:23 smithi026 ceph-mon[92215]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded) 2024-04-16T07:29:24.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:23 smithi104 ceph-mon[81027]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded) 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 29) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:24.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.107+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:24.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.108+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 34 ==== client_session(renewcaps seq 29) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:29:24.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:24.108+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 29) v1 from mds.0 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:24.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:24.393+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:25.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:24 smithi026 ceph-mon[92215]: pgmap v170: 65 pgs: 7 peering, 18 active+undersized, 40 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:24 smithi104 ceph-mon[81027]: pgmap v170: 65 pgs: 7 peering, 18 active+undersized, 40 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:25.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:25.108+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 230 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:25.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.393+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:25.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101769 2024-04-16T07:29:25.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 466, success, outsize: 120 2024-04-16T07:29:25.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.486+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.486+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.486+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.486+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.486+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101858 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 468, success, outsize: 120 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.504+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.504+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.504+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.504+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.504+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101950 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 470, success, outsize: 120 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.524+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.524+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.524+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.524+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.524+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102048 2024-04-16T07:29:25.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 472, success, outsize: 120 2024-04-16T07:29:25.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.551+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.551+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.551+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.551+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.551+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102143 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 474, success, outsize: 120 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.577+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.577+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.577+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.577+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.577+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102232 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 476, success, outsize: 120 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.600+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.600+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.600+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.600+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.600+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102324 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 478, success, outsize: 120 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.624+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.624+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.624+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.624+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.625+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102422 2024-04-16T07:29:25.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 480, success, outsize: 120 2024-04-16T07:29:25.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.650+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.650+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.650+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.650+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.651+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102517 2024-04-16T07:29:25.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 482, success, outsize: 120 2024-04-16T07:29:25.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.675+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.675+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.675+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.675+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.675+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102606 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 484, success, outsize: 120 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.696+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.696+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.696+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.696+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.696+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102698 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 486, success, outsize: 120 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.719+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.719+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.719+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.719+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.719+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102796 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 488, success, outsize: 120 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.744+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.744+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.744+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.744+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.744+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102891 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 490, success, outsize: 120 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.768+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.768+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.768+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.768+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.768+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102980 2024-04-16T07:29:25.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 492, success, outsize: 120 2024-04-16T07:29:25.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.791+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.791+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.791+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.791+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.791+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103072 2024-04-16T07:29:25.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 494, success, outsize: 120 2024-04-16T07:29:25.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.816+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.816+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.816+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.816+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.816+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103170 2024-04-16T07:29:25.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 496, success, outsize: 120 2024-04-16T07:29:25.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.843+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.843+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.843+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.843+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.843+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.869 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103265 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.871+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 498, success, outsize: 120 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.871+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.871+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.871+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.871+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103354 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 500, success, outsize: 120 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.897+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.897+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.897+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.897+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.897+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.923 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103446 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 502, success, outsize: 120 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.925+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.925+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.925+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.925+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.925+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103544 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 504, success, outsize: 120 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.950+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.950+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.950+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.950+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.950+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103639 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 506, success, outsize: 120 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.975+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.975+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.975+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.975+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.975+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:25.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103728 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 508, success, outsize: 120 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.998+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.998+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.998+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.998+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:25.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:25.998+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103820 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 510, success, outsize: 120 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.020+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.020+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.020+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.020+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.020+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103918 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 512, success, outsize: 120 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.041+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.041+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.041+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.041+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.041+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104013 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 514, success, outsize: 120 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.064+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.064+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.064+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.064+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.064+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104102 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 516, success, outsize: 120 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.087+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.087+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.087+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.087+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.087+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104194 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 518, success, outsize: 120 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.107+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.107+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.107+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.107+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.107+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:26.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:26.108+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104292 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.128+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 520, success, outsize: 120 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.128+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.128+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.128+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:29:26.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.128+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:29:26.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:26.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:26.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:26.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:26.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:26.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:26.393+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:26 smithi104 ceph-mon[81027]: pgmap v171: 65 pgs: 7 peering, 58 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:27.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:26 smithi026 ceph-mon[92215]: pgmap v171: 65 pgs: 7 peering, 58 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:27.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:27.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:27.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:27.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:27.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:27.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:27.108+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:27.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:27.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:27.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:27.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:27.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:27.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:27.393+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:28.108+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:28.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:28.394+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: pgmap v172: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all osd 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T07:29:28.490 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T07:29:28.491 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T07:29:28.491 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T07:29:28.491 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T07:29:28.491 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T07:29:28.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: pgmap v172: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all osd 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T07:29:28.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T07:29:28.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:29.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:29.109+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:29.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:29.394+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:29.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T07:29:29.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:29:29.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T07:29:29.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T07:29:29.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi026.kqpyuh"]}]: dispatch 2024-04-16T07:29:29.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:29 smithi026 ceph-mon[92215]: Upgrade: It appears safe to stop mds.cephfs.smithi026.kqpyuh 2024-04-16T07:29:29.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T07:29:29.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:29:29.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T07:29:29.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T07:29:29.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi026.kqpyuh"]}]: dispatch 2024-04-16T07:29:29.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:29 smithi104 ceph-mon[81027]: Upgrade: It appears safe to stop mds.cephfs.smithi026.kqpyuh 2024-04-16T07:29:30.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:30.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:30.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:30.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:30.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:30.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:30.109+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:30.388 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:30 smithi026 ceph-mon[92215]: pgmap v174: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:30.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:30.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:30.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:30 smithi104 ceph-mon[81027]: pgmap v174: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:31.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:31.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:31.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:31.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:31.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:31.109+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:31.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.338+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004330 con 0x55fc3f5009a0 2024-04-16T07:29:31.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.340+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 5 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a0004330 con 0x55fc3f5009a0 2024-04-16T07:29:31.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:31.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:31.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:31.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:31.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:31.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:31.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:31.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:31 smithi026 ceph-mon[92215]: Upgrade: Updating mds.cephfs.smithi026.kqpyuh 2024-04-16T07:29:31.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:31 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:31.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:31 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:29:31.967 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:31 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:31.968 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:31 smithi026 ceph-mon[92215]: Deploying daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:29:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:31 smithi104 ceph-mon[81027]: Upgrade: Updating mds.cephfs.smithi026.kqpyuh 2024-04-16T07:29:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:31 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:31 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kqpyuh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:29:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:31 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:31 smithi104 ceph-mon[81027]: Deploying daemon mds.cephfs.smithi026.kqpyuh on smithi026 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:32.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:32.109+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 29) v5 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:29:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:32.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:32.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 33 ==== client_session(renewcaps seq 29) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:32.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:32.394+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 29) v1 from mds.0 2024-04-16T07:29:33.076 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:32 smithi026 ceph-mon[92215]: pgmap v175: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:33.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:32 smithi104 ceph-mon[81027]: pgmap v175: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.109+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.109+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.109+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.110+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.110+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:33.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:33.110+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:33.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:33.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:33.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:33.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:33.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:33.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:33.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:34.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:34.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:34.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:34.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:34.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:34.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:34.110+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:34.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:34.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:34.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:34.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:34.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:34.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:34.394+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:34 smithi104 ceph-mon[81027]: pgmap v176: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:34 smithi104 ceph-mon[81027]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T07:29:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:35.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:35.111+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:34 smithi026 ceph-mon[92215]: pgmap v176: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:34 smithi026 ceph-mon[92215]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T07:29:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:35.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:35.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:35.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:35.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:35.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:35.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:35.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:36.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:35 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 2 up:standby 2024-04-16T07:29:36.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:29:36.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:36.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:36.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:36.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:36.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:36.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:36.111+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:36.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:35 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 2 up:standby 2024-04-16T07:29:36.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:36.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:36.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:36.940 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:36 smithi026 ceph-mon[92215]: pgmap v178: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:37.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:36 smithi104 ceph-mon[81027]: pgmap v178: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:37.111+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:37.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:37.396+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:38.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:38.111+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:38.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:38.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:38.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:38.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:38.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:38.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:38.396+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:39.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:38 smithi026 ceph-mon[92215]: pgmap v179: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:39.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:38 smithi104 ceph-mon[81027]: pgmap v179: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.111+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.111+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.111+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.111+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.112+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:39.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:39.112+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:39.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:39.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:39.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:39.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:39.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:39.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:39.395+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:40.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:40.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:40.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:40.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:40.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:40.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:40.112+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:40.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:40.397+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: pgmap v180: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:boot 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:29:40.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:40 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: pgmap v180: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:boot 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kwjewe=up:active} 3 up:standby 2024-04-16T07:29:40.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:40 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:29:41.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:41.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:41.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:41.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:41.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:41.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:41.112+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:41.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:41.397+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:42.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:42.112+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:42.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:42.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:42.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:42.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:42.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:42.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:42.397+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:42.729 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:42 smithi026 ceph-mon[92215]: pgmap v181: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:42 smithi104 ceph-mon[81027]: pgmap v181: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:43.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:43.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.396+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.396+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.396+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.397+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:43.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:43.397+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:29:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:29:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:29:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 30) v5 -- 0x7fbf68004560 con 0x563ccaa1cfa0 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6828/1284848687 35 ==== client_session(renewcaps seq 30) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004720 con 0x563ccaa1cfa0 2024-04-16T07:29:44.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:44.113+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 30) v1 from mds.0 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:44.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:44.398+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:44.753 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:44 smithi026 ceph-mon[92215]: pgmap v182: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:44 smithi104 ceph-mon[81027]: pgmap v182: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:45.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:45.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:45.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:45.398+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:46.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:46.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:46.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:46.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:46.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:46.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:46.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:46.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:46.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:46.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:46.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:46.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:46.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:46.398+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:46.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:46 smithi026 ceph-mon[92215]: pgmap v183: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:46.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:46 smithi104 ceph-mon[81027]: pgmap v183: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:47.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:47.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:47.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:47.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:47.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:47.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:47.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:47.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:47.399+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:48.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:47 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:48.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:47 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:48.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:47 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:48.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:47 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:48.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:48.113+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:48.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:48.399+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:48.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 -- 172.21.15.26:0/3624134911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fbd40 msgr2=0x7f56a40fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:48.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 --2- 172.21.15.26:0/3624134911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fbd40 0x7f56a40fc150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5694004940 tx=0x7f5694009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:48.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 -- 172.21.15.26:0/3624134911 shutdown_connections 2024-04-16T07:29:48.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 --2- 172.21.15.26:0/3624134911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fcff0 0x7f56a40fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:48.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 --2- 172.21.15.26:0/3624134911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fbd40 0x7f56a40fc150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:48.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 -- 172.21.15.26:0/3624134911 >> 172.21.15.26:0/3624134911 conn(0x7f56a40f74c0 msgr2=0x7f56a40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:48.950 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:48 smithi026 ceph-mon[92215]: pgmap v184: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:48.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 -- 172.21.15.26:0/3624134911 shutdown_connections 2024-04-16T07:29:48.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.950+0000 7f56ab000700 1 -- 172.21.15.26:0/3624134911 wait complete. 2024-04-16T07:29:48.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.951+0000 7f56ab000700 1 Processor -- start 2024-04-16T07:29:48.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.951+0000 7f56ab000700 1 -- start start 2024-04-16T07:29:48.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f56ab000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fbd40 0x7f56a40ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f56ab000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f56ab000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40fe870 con 0x7f56a40fcff0 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f56ab000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40fe9b0 con 0x7f56a40fbd40 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f569b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f569b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37906/0 (socket says 172.21.15.26:37906) 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f569b400700 1 -- 172.21.15.26:0/39066972 learned_addr learned my addr 172.21.15.26:0/39066972 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:48.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.952+0000 7f56a3e00700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fbd40 0x7f56a40ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f569b400700 1 -- 172.21.15.26:0/39066972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fbd40 msgr2=0x7f56a40ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f569b400700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fbd40 0x7f56a40ffdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f569b400700 1 -- 172.21.15.26:0/39066972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694005040 con 0x7f56a40fcff0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f569b400700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c8c0 tx=0x7f568c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568c00a760 con 0x7f56a40fcff0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a40fec90 con 0x7f56a40fcff0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f568c00ad90 con 0x7f56a40fcff0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.953+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568c0113a0 con 0x7f56a40fcff0 2024-04-16T07:29:48.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.954+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a4115b00 con 0x7f56a40fcff0 2024-04-16T07:29:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.955+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a4101b80 con 0x7f56a40fcff0 2024-04-16T07:29:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.956+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f568c012400 con 0x7f56a40fcff0 2024-04-16T07:29:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.957+0000 7f56a0c00700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 0x7f56880710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.957+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f568c08e870 con 0x7f56a40fcff0 2024-04-16T07:29:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.957+0000 7f56a3e00700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 0x7f56880710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.958+0000 7f56a3e00700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 0x7f56880710c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:48.964+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f568c0576a0 con 0x7f56a40fcff0 2024-04-16T07:29:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:48 smithi104 ceph-mon[81027]: pgmap v184: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.113+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.113+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.113+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.114+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.114+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:49.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:49.114+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:49.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.224+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a405e400 con 0x7f568806ec10 2024-04-16T07:29:49.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.226+0000 7f56a0c00700 1 -- 172.21.15.26:0/39066972 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f56a405e400 con 0x7f568806ec10 2024-04-16T07:29:49.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 msgr2=0x7f56880710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 0x7f56880710c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:49.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 msgr2=0x7f56a40fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c8c0 tx=0x7f568c0044c0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 shutdown_connections 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f568806ec10 0x7f56880710c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56a40fbd40 0x7f56a40ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 --2- 172.21.15.26:0/39066972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a40fcff0 0x7f56a40fe330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 >> 172.21.15.26:0/39066972 conn(0x7f56a40f74c0 msgr2=0x7f56a4065b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 shutdown_connections 2024-04-16T07:29:49.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.230+0000 7f56ab000700 1 -- 172.21.15.26:0/39066972 wait complete. 2024-04-16T07:29:49.245 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.378+0000 7f09aca00700 1 -- 172.21.15.26:0/232133122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8071e50 msgr2=0x7f09a8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.378+0000 7f09aca00700 1 --2- 172.21.15.26:0/232133122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8071e50 0x7f09a8070350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0998004940 tx=0x7f0998009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 -- 172.21.15.26:0/232133122 shutdown_connections 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 --2- 172.21.15.26:0/232133122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09a8070890 0x7f09a8070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 --2- 172.21.15.26:0/232133122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8071e50 0x7f09a8070350 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 -- 172.21.15.26:0/232133122 >> 172.21.15.26:0/232133122 conn(0x7f09a80f8540 msgr2=0x7f09a80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 -- 172.21.15.26:0/232133122 shutdown_connections 2024-04-16T07:29:49.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 -- 172.21.15.26:0/232133122 wait complete. 2024-04-16T07:29:49.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.379+0000 7f09aca00700 1 Processor -- start 2024-04-16T07:29:49.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09aca00700 1 -- start start 2024-04-16T07:29:49.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09aca00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09aca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09a8071e50 0x7f09a8110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09aca00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8110860 con 0x7f09a8070890 2024-04-16T07:29:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09aca00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8110fa0 con 0x7f09a8071e50 2024-04-16T07:29:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09a5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09a5e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37912/0 (socket says 172.21.15.26:37912) 2024-04-16T07:29:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.380+0000 7f09a5e00700 1 -- 172.21.15.26:0/1852149770 learned_addr learned my addr 172.21.15.26:0/1852149770 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f09a5e00700 1 -- 172.21.15.26:0/1852149770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09a8071e50 msgr2=0x7f09a8110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f09a5e00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09a8071e50 0x7f09a8110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f09a5e00700 1 -- 172.21.15.26:0/1852149770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998005040 con 0x7f09a8070890 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f09a5e00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0998000c00 tx=0x7f0998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099800e040 con 0x7f09a8070890 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f099800bcc0 con 0x7f09a8070890 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8111220 con 0x7f09a8070890 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.381+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099801c630 con 0x7f09a8070890 2024-04-16T07:29:49.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.382+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a81117e0 con 0x7f09a8070890 2024-04-16T07:29:49.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.383+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a805f5f0 con 0x7f09a8070890 2024-04-16T07:29:49.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.384+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0998013420 con 0x7f09a8070890 2024-04-16T07:29:49.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.385+0000 7f099e000700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 0x7f098c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.385+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f099808fd10 con 0x7f09a8070890 2024-04-16T07:29:49.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.385+0000 7f09a5400700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 0x7f098c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:49.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.386+0000 7f09a5400700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 0x7f098c0710b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0994004ad0 tx=0x7f0994007d40 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:49.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.390+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f099805c5f0 con 0x7f09a8070890 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:49.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:49.399+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:49.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.661+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09a8111bd0 con 0x7f098c06ec00 2024-04-16T07:29:49.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.663+0000 7f099e000700 1 -- 172.21.15.26:0/1852149770 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f09a8111bd0 con 0x7f098c06ec00 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 msgr2=0x7f098c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 0x7f098c0710b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0994004ad0 tx=0x7f0994007d40 comp rx=0 tx=0).stop 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 msgr2=0x7f09a810fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0998000c00 tx=0x7f0998009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 shutdown_connections 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f098c06ec00 0x7f098c0710b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a8070890 0x7f09a810fd20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 --2- 172.21.15.26:0/1852149770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09a8071e50 0x7f09a8110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 >> 172.21.15.26:0/1852149770 conn(0x7f09a80f8540 msgr2=0x7f09a80f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:49.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 shutdown_connections 2024-04-16T07:29:49.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.666+0000 7f09aca00700 1 -- 172.21.15.26:0/1852149770 wait complete. 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.812+0000 7f40c9400700 1 -- 172.21.15.26:0/1774656806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c4100070 msgr2=0x7f40c41004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.812+0000 7f40c9400700 1 --2- 172.21.15.26:0/1774656806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c4100070 0x7f40c41004e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f40ac004940 tx=0x7f40ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 -- 172.21.15.26:0/1774656806 shutdown_connections 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 --2- 172.21.15.26:0/1774656806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c4100070 0x7f40c41004e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 --2- 172.21.15.26:0/1774656806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c40fef00 0x7f40c40ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 -- 172.21.15.26:0/1774656806 >> 172.21.15.26:0/1774656806 conn(0x7f40c40fa790 msgr2=0x7f40c40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:49.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 -- 172.21.15.26:0/1774656806 shutdown_connections 2024-04-16T07:29:49.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 -- 172.21.15.26:0/1774656806 wait complete. 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.813+0000 7f40c9400700 1 Processor -- start 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c9400700 1 -- start start 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c9400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c4100070 0x7f40c410ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c9400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4110570 con 0x7f40c40fef00 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c9400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4110dc0 con 0x7f40c4100070 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c3e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37918/0 (socket says 172.21.15.26:37918) 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.814+0000 7f40c3e00700 1 -- 172.21.15.26:0/2897296353 learned_addr learned my addr 172.21.15.26:0/2897296353 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40bbe00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c4100070 0x7f40c410ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40c3e00700 1 -- 172.21.15.26:0/2897296353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c4100070 msgr2=0x7f40c410ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:49.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40c3e00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c4100070 0x7f40c410ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:49.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40c3e00700 1 -- 172.21.15.26:0/2897296353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac005040 con 0x7f40c40fef00 2024-04-16T07:29:49.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40c3e00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40b4002a50 tx=0x7f40b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:49.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.815+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40b4007cc0 con 0x7f40c40fef00 2024-04-16T07:29:49.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.816+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40b400f040 con 0x7f40c40fef00 2024-04-16T07:29:49.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.816+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40b4009640 con 0x7f40c40fef00 2024-04-16T07:29:49.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.816+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c41110a0 con 0x7f40c40fef00 2024-04-16T07:29:49.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.816+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c41115e0 con 0x7f40c40fef00 2024-04-16T07:29:49.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.817+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c40fe290 con 0x7f40c40fef00 2024-04-16T07:29:49.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.818+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40b40097a0 con 0x7f40c40fef00 2024-04-16T07:29:49.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.819+0000 7f40c0c00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 0x7f40a80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:49.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.819+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f40b408e630 con 0x7f40c40fef00 2024-04-16T07:29:49.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.819+0000 7f40bbe00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 0x7f40a80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:49.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.820+0000 7f40bbe00700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 0x7f40a80710b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40ac004f90 tx=0x7f40ac009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:49.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:49.824+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f40b405af10 con 0x7f40c40fef00 2024-04-16T07:29:50.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:50.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:50.000 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:50.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.070+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f40c41119d0 con 0x7f40a806ec00 2024-04-16T07:29:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.099+0000 7f40c0c00700 1 -- 172.21.15.26:0/2897296353 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f40c41119d0 con 0x7f40a806ec00 2024-04-16T07:29:50.099 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:29:50.099 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (6m) 3s ago 17m 14.0M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (3m) 3s ago 17m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (3m) 30s ago 16m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (6m) 3s ago 16m 73.0M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (11s) 3s ago 10m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (9m) 3s ago 9m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 91647b42c5c5 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (10m) 30s ago 10m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (9m) 30s ago 9m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (6m) 3s ago 18m 508M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (5m) 30s ago 15m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (5m) 3s ago 19m 47.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (4m) 30s ago 15m 36.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (6m) 3s ago 17m 7964k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 30s ago 15m 8070k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (3m) 3s ago 14m 51.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:29:50.100 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (2m) 3s ago 14m 51.4M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:29:50.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (2m) 3s ago 13m 42.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:29:50.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (90s) 30s ago 13m 50.9M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:29:50.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (62s) 30s ago 12m 41.4M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:29:50.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (35s) 30s ago 11m 11.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:29:50.101 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (5m) 3s ago 16m 60.0M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 msgr2=0x7f40a80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 0x7f40a80710b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40ac004f90 tx=0x7f40ac009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 msgr2=0x7f40c410fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40b4002a50 tx=0x7f40b40046b0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 shutdown_connections 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f40a806ec00 0x7f40a80710b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40c40fef00 0x7f40c410fa30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 --2- 172.21.15.26:0/2897296353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40c4100070 0x7f40c410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 >> 172.21.15.26:0/2897296353 conn(0x7f40c40fa790 msgr2=0x7f40c40fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.103+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 shutdown_connections 2024-04-16T07:29:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.104+0000 7f40c9400700 1 -- 172.21.15.26:0/2897296353 wait complete. 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:50.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:50.114+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 -- 172.21.15.26:0/680803051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fcff0 msgr2=0x7fb3740fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 --2- 172.21.15.26:0/680803051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fcff0 0x7fb3740fd460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb360004940 tx=0x7fb360009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 -- 172.21.15.26:0/680803051 shutdown_connections 2024-04-16T07:29:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 --2- 172.21.15.26:0/680803051 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fcff0 0x7fb3740fd460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 --2- 172.21.15.26:0/680803051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fbd40 0x7fb3740fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 -- 172.21.15.26:0/680803051 >> 172.21.15.26:0/680803051 conn(0x7fb3740f74c0 msgr2=0x7fb3740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 -- 172.21.15.26:0/680803051 shutdown_connections 2024-04-16T07:29:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.260+0000 7fb378e00700 1 -- 172.21.15.26:0/680803051 wait complete. 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 Processor -- start 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 -- start start 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fcff0 0x7fb374110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374110820 con 0x7fb3740fbd40 2024-04-16T07:29:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.261+0000 7fb378e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374110fc0 con 0x7fb3740fcff0 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37928/0 (socket says 172.21.15.26:37928) 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 -- 172.21.15.26:0/4132166620 learned_addr learned my addr 172.21.15.26:0/4132166620 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371400700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fcff0 0x7fb374110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 -- 172.21.15.26:0/4132166620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fcff0 msgr2=0x7fb374110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fcff0 0x7fb374110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 -- 172.21.15.26:0/4132166620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb360005040 con 0x7fb3740fbd40 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.262+0000 7fb371e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b790 tx=0x7fb364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.263+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36400f770 con 0x7fb3740fbd40 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.263+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36400fda0 con 0x7fb3740fbd40 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.263+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3741112a0 con 0x7fb3740fbd40 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.263+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36400d310 con 0x7fb3740fbd40 2024-04-16T07:29:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.263+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb374111860 con 0x7fb3740fbd40 2024-04-16T07:29:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.264+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb374101980 con 0x7fb3740fbd40 2024-04-16T07:29:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.265+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb364010420 con 0x7fb3740fbd40 2024-04-16T07:29:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.267+0000 7fb36a000700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 0x7fb3580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.267+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb36408e870 con 0x7fb3740fbd40 2024-04-16T07:29:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.267+0000 7fb371400700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 0x7fb3580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:50.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.268+0000 7fb371400700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 0x7fb3580710c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb36000b3b0 tx=0x7fb360009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:50.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.272+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb36405b1d0 con 0x7fb3740fbd40 2024-04-16T07:29:50.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.399+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.399+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.399+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.399+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:50.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:50.400+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:50.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.598+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3741049d0 con 0x7fb3740fbd40 2024-04-16T07:29:50.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.600+0000 7fb36a000700 1 -- 172.21.15.26:0/4132166620 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fb3741049d0 con 0x7fb3740fbd40 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T07:29:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 11 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:29:50.601 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 msgr2=0x7fb3580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 0x7fb3580710c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb36000b3b0 tx=0x7fb360009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 msgr2=0x7fb37410fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b790 tx=0x7fb364009250 comp rx=0 tx=0).stop 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 shutdown_connections 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb35806ec10 0x7fb3580710c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3740fbd40 0x7fb37410fc50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 --2- 172.21.15.26:0/4132166620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3740fcff0 0x7fb374110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 >> 172.21.15.26:0/4132166620 conn(0x7fb3740f74c0 msgr2=0x7fb374065b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 shutdown_connections 2024-04-16T07:29:50.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.603+0000 7fb378e00700 1 -- 172.21.15.26:0/4132166620 wait complete. 2024-04-16T07:29:50.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 -- 172.21.15.26:0/2494207883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 msgr2=0x7f19340ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 --2- 172.21.15.26:0/2494207883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340ffa30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1928004940 tx=0x7f1928009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 -- 172.21.15.26:0/2494207883 shutdown_connections 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 --2- 172.21.15.26:0/2494207883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f19340fff70 0x7f1934102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 --2- 172.21.15.26:0/2494207883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340ffa30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 -- 172.21.15.26:0/2494207883 >> 172.21.15.26:0/2494207883 conn(0x7f19340f74c0 msgr2=0x7f19340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 -- 172.21.15.26:0/2494207883 shutdown_connections 2024-04-16T07:29:50.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.764+0000 7f193c400700 1 -- 172.21.15.26:0/2494207883 wait complete. 2024-04-16T07:29:50.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 Processor -- start 2024-04-16T07:29:50.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 -- start start 2024-04-16T07:29:50.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f19340fff70 0x7f19340fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340fbee0 con 0x7f19340662f0 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.765+0000 7f193c400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340fc050 con 0x7f19340fff70 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37940/0 (socket says 172.21.15.26:37940) 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 -- 172.21.15.26:0/3172117307 learned_addr learned my addr 172.21.15.26:0/3172117307 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 -- 172.21.15.26:0/3172117307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f19340fff70 msgr2=0x7f19340fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f19340fff70 0x7f19340fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 -- 172.21.15.26:0/3172117307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928005040 con 0x7f19340662f0 2024-04-16T07:29:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.766+0000 7f1939a00700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.767+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192800e040 con 0x7f19340662f0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.767+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f192800bcc0 con 0x7f19340662f0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.767+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19340fc2d0 con 0x7f19340662f0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.767+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192801c630 con 0x7f19340662f0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.767+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934115a90 con 0x7f19340662f0 2024-04-16T07:29:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.768+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193405f5f0 con 0x7f19340662f0 2024-04-16T07:29:50.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.769+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1928013980 con 0x7f19340662f0 2024-04-16T07:29:50.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.770+0000 7f1931600700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 0x7f191c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:50.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.770+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f192808fbc0 con 0x7f19340662f0 2024-04-16T07:29:50.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.770+0000 7f1939000700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 0x7f191c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:50.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.771+0000 7f1939000700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 0x7f191c0710b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1920004b70 tx=0x7f1920007de0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:50.775+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f19280589f0 con 0x7f19340662f0 2024-04-16T07:29:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.061+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1934115d80 con 0x7f19340662f0 2024-04-16T07:29:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.062+0000 7f1931600700 1 -- 172.21.15.26:0/3172117307 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f192805c0b0 con 0x7f19340662f0 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:e18 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:29:51.062 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 15 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:19:59.548919+0000 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:29:51.063 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14470} 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 0 members: 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{0:14470} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:51.064 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:29:51.065 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:29:51.065 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:24287} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:51.065 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:29:51.065 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:34404} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 msgr2=0x7f191c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 0x7f191c0710b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1920004b70 tx=0x7f1920007de0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 msgr2=0x7f19340fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 shutdown_connections 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f191c06ec00 0x7f191c0710b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19340662f0 0x7f19340fd460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 --2- 172.21.15.26:0/3172117307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f19340fff70 0x7f19340fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 >> 172.21.15.26:0/3172117307 conn(0x7f19340f74c0 msgr2=0x7f19340f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 shutdown_connections 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.065+0000 7f193c400700 1 -- 172.21.15.26:0/3172117307 wait complete. 2024-04-16T07:29:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2024-04-16T07:29:51.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:51.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:51.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:51.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:51.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:51.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:51.114+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 -- 172.21.15.26:0/649674373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac1002a0 msgr2=0x7fb3ac100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 --2- 172.21.15.26:0/649674373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac1002a0 0x7fb3ac100710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb398004940 tx=0x7fb398009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 -- 172.21.15.26:0/649674373 shutdown_connections 2024-04-16T07:29:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 --2- 172.21.15.26:0/649674373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac1002a0 0x7fb3ac100710 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 --2- 172.21.15.26:0/649674373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac0ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 -- 172.21.15.26:0/649674373 >> 172.21.15.26:0/649674373 conn(0x7fb3ac0fa790 msgr2=0x7fb3ac0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:51.217 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:50 smithi026 ceph-mon[92215]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:51.217 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:50 smithi026 ceph-mon[92215]: pgmap v185: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:51.217 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:50 smithi026 ceph-mon[92215]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:51.217 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:50 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4132166620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 -- 172.21.15.26:0/649674373 shutdown_connections 2024-04-16T07:29:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.217+0000 7fb3b4200700 1 -- 172.21.15.26:0/649674373 wait complete. 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.218+0000 7fb3b4200700 1 Processor -- start 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.218+0000 7fb3b4200700 1 -- start start 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b4200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b4200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3ac1002a0 0x7fb3ac110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b4200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac110640 con 0x7fb3ac0feff0 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b4200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac1107b0 con 0x7fb3ac1002a0 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37950/0 (socket says 172.21.15.26:37950) 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 -- 172.21.15.26:0/270127770 learned_addr learned my addr 172.21.15.26:0/270127770 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 -- 172.21.15.26:0/270127770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3ac1002a0 msgr2=0x7fb3ac110040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3ac1002a0 0x7fb3ac110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.219+0000 7fb3b1800700 1 -- 172.21.15.26:0/270127770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398005040 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3b1800700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0002a50 tx=0x7fb3a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a0007cc0 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac115210 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3a000f040 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a0009640 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.220+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac1157d0 con 0x7fb3ac0feff0 2024-04-16T07:29:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.221+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac04c710 con 0x7fb3ac0feff0 2024-04-16T07:29:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.223+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb3a00097a0 con 0x7fb3ac0feff0 2024-04-16T07:29:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.224+0000 7fb3a9600700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 0x7fb3940710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.224+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3a008e680 con 0x7fb3ac0feff0 2024-04-16T07:29:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.224+0000 7fb3b0e00700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 0x7fb3940710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.225+0000 7fb3b0e00700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 0x7fb3940710b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb398004f90 tx=0x7fb398009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.229+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb3a005af60 con 0x7fb3ac0feff0 2024-04-16T07:29:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:50 smithi104 ceph-mon[81027]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:50 smithi104 ceph-mon[81027]: pgmap v185: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:50 smithi104 ceph-mon[81027]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:50 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4132166620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:51.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:51.400+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.483+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3ac1045c0 con 0x7fb39406ec00 2024-04-16T07:29:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.485+0000 7fb3a9600700 1 -- 172.21.15.26:0/270127770 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb3ac1045c0 con 0x7fb39406ec00 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "13/21 daemons upgraded", 2024-04-16T07:29:51.485 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T07:29:51.486 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:29:51.486 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 msgr2=0x7fb3940710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 0x7fb3940710b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb398004f90 tx=0x7fb398009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 msgr2=0x7fb3ac10fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0002a50 tx=0x7fb3a00046b0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 shutdown_connections 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb39406ec00 0x7fb3940710b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3ac0feff0 0x7fb3ac10fb00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.487+0000 7fb3b4200700 1 --2- 172.21.15.26:0/270127770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3ac1002a0 0x7fb3ac110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.488+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 >> 172.21.15.26:0/270127770 conn(0x7fb3ac0fa790 msgr2=0x7fb3ac0fbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.488+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 shutdown_connections 2024-04-16T07:29:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.488+0000 7fb3b4200700 1 -- 172.21.15.26:0/270127770 wait complete. 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1625603325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0662f0 msgr2=0x7fe9ac0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1625603325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0662f0 0x7fe9ac0ffa30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe99c004940 tx=0x7fe99c009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1625603325 shutdown_connections 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1625603325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1625603325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0662f0 0x7fe9ac0ffa30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.638+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1625603325 >> 172.21.15.26:0/1625603325 conn(0x7fe9ac0f74c0 msgr2=0x7fe9ac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.639+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1625603325 shutdown_connections 2024-04-16T07:29:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.639+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1625603325 wait complete. 2024-04-16T07:29:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.639+0000 7fe9b0c00700 1 Processor -- start 2024-04-16T07:29:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9b0c00700 1 -- start start 2024-04-16T07:29:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9b0c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9ac0662f0 0x7fe9ac10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9b0c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9b0c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac110790 con 0x7fe9ac0fff70 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9b0c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac110f90 con 0x7fe9ac0662f0 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9a9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9a9400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37966/0 (socket says 172.21.15.26:37966) 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.640+0000 7fe9a9400700 1 -- 172.21.15.26:0/1367370850 learned_addr learned my addr 172.21.15.26:0/1367370850 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a9400700 1 -- 172.21.15.26:0/1367370850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9ac0662f0 msgr2=0x7fe9ac10fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a9400700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9ac0662f0 0x7fe9ac10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a9400700 1 -- 172.21.15.26:0/1367370850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c005040 con 0x7fe9ac0fff70 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a9400700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe99800eb40 tx=0x7fe99800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe998007cc0 con 0x7fe9ac0fff70 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe99800f040 con 0x7fe9ac0fff70 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe998009620 con 0x7fe9ac0fff70 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.641+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac111210 con 0x7fe9ac0fff70 2024-04-16T07:29:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.642+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac111750 con 0x7fe9ac0fff70 2024-04-16T07:29:51.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.642+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac05f5f0 con 0x7fe9ac0fff70 2024-04-16T07:29:51.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.644+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe998009780 con 0x7fe9ac0fff70 2024-04-16T07:29:51.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.644+0000 7fe9a2000700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 0x7fe9900710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:29:51.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.645+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe99808e4a0 con 0x7fe9ac0fff70 2024-04-16T07:29:51.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.645+0000 7fe9a9e00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 0x7fe9900710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:29:51.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.645+0000 7fe9a9e00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 0x7fe9900710c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe99c013870 tx=0x7fe99c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:29:51.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.649+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe99805aeb0 con 0x7fe9ac0fff70 2024-04-16T07:29:51.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.982+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe9ac04bed0 con 0x7fe9ac0fff70 2024-04-16T07:29:51.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.983+0000 7fe9a2000700 1 -- 172.21.15.26:0/1367370850 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe99805aa40 con 0x7fe9ac0fff70 2024-04-16T07:29:51.984 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:29:51.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 msgr2=0x7fe9900710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 0x7fe9900710c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe99c013870 tx=0x7fe99c009e30 comp rx=0 tx=0).stop 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 msgr2=0x7fe9ac110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe99800eb40 tx=0x7fe99800b040 comp rx=0 tx=0).stop 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 shutdown_connections 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe99006ec10 0x7fe9900710c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9ac0662f0 0x7fe9ac10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 --2- 172.21.15.26:0/1367370850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9ac0fff70 0x7fe9ac110190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 >> 172.21.15.26:0/1367370850 conn(0x7fe9ac0f74c0 msgr2=0x7fe9ac0fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 shutdown_connections 2024-04-16T07:29:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:29:51.986+0000 7fe9b0c00700 1 -- 172.21.15.26:0/1367370850 wait complete. 2024-04-16T07:29:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:51 smithi026 ceph-mon[92215]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:52.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:51 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3172117307' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:29:52.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:52.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:52.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:52.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 174 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:52.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:52.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:52.114+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:51 smithi104 ceph-mon[81027]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:51 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3172117307' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_session(request_renewcaps seq 30) v5 -- 0x7fe9ac0040b0 con 0x55fc3f51baa0 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:52.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004820 con 0x55fc3f51baa0 2024-04-16T07:29:52.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:52.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.400+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:52.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.401+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6828/1284848687 34 ==== client_session(renewcaps seq 30) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:52.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:52.401+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 30) v1 from mds.0 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119446 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 354, success, outsize: 120 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.095+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.095+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.095+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.095+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.095+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:53.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:53.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:53.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 175 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:53.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:53.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.114+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:53.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119528 2024-04-16T07:29:53.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 356, success, outsize: 120 2024-04-16T07:29:53.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.116+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.116+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.116+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.116+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.116+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119613 2024-04-16T07:29:53.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 358, success, outsize: 120 2024-04-16T07:29:53.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.138+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.138+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.138+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.138+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.138+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119704 2024-04-16T07:29:53.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 360, success, outsize: 120 2024-04-16T07:29:53.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.163+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.163+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.163+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.163+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.163+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119792 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 362, success, outsize: 120 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.190+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.190+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.190+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.190+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.190+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119874 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 364, success, outsize: 120 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.214+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.214+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.214+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.214+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.214+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119959 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 366, success, outsize: 120 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.240+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.240+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.240+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.240+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.240+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:52 smithi026 ceph-mon[92215]: pgmap v186: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:53.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:52 smithi026 ceph-mon[92215]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:53.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:52 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1367370850' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120050 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 368, success, outsize: 120 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.268+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.268+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.268+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.268+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.268+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120138 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 370, success, outsize: 120 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.294+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.294+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.294+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.294+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.294+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120220 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 372, success, outsize: 120 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.317+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.317+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.317+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.317+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.317+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:52 smithi104 ceph-mon[81027]: pgmap v186: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:53.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:52 smithi104 ceph-mon[81027]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:29:53.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:52 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1367370850' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:29:53.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120305 2024-04-16T07:29:53.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 374, success, outsize: 120 2024-04-16T07:29:53.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.342+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.342+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.342+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.342+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.342+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120396 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 376, success, outsize: 120 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.365+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.365+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.365+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.365+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.365+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120484 2024-04-16T07:29:53.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 378, success, outsize: 120 2024-04-16T07:29:53.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.388+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.388+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.388+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.388+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.388+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:53.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:53.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:53.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:53.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:53.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:53.400+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120566 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 380, success, outsize: 120 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.410+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.410+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.410+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.410+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.410+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120651 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 382, success, outsize: 120 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.427+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.427+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.427+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.427+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.427+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120742 2024-04-16T07:29:53.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 384, success, outsize: 120 2024-04-16T07:29:53.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.447+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.447+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.447+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.447+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.447+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120830 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 386, success, outsize: 120 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.466+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.466+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.466+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.466+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.466+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120912 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.483+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 388, success, outsize: 120 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.483+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.483+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.483+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.483+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120997 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 390, success, outsize: 120 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.504+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.504+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.504+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.504+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.504+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121088 2024-04-16T07:29:53.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 392, success, outsize: 120 2024-04-16T07:29:53.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.524+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.524+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.524+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.524+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.524+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121176 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 394, success, outsize: 120 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.545+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.545+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.545+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.545+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.545+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121258 2024-04-16T07:29:53.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 396, success, outsize: 120 2024-04-16T07:29:53.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.565+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.565+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.565+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.565+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.565+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121343 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 398, success, outsize: 120 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.588+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.588+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.588+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.588+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.588+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121434 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 400, success, outsize: 120 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.611+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.611+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.611+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.611+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.611+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121522 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 402, success, outsize: 120 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.635+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.635+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.635+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.635+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.635+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121604 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 404, success, outsize: 120 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.658+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.658+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.658+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.658+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.658+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121689 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 406, success, outsize: 120 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.682+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.682+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.682+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.682+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.682+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:53.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121780 2024-04-16T07:29:53.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 408, success, outsize: 120 2024-04-16T07:29:53.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.709+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:29:53.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.709+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:29:53.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.709+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:29:53.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.709+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:29:53.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:53.709+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:54.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:54.115+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:54.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:54.401+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: pgmap v187: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:54 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:55.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:55.115+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: pgmap v187: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:29:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:54 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:55.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:55.401+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:56.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:56.115+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: Detected new or changed devices on smithi026 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.220 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:55 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi026.kwjewe"]}]: dispatch 2024-04-16T07:29:56.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: Detected new or changed devices on smithi026 2024-04-16T07:29:56.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:29:56.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:29:56.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:55 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi026.kwjewe"]}]: dispatch 2024-04-16T07:29:56.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:56.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:56.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:56.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:56.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:56.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:56.401+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:57.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:57.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:57.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:57.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:57.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:57.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:57.115+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:57.230 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:56 smithi026 ceph-mon[92215]: Upgrade: It appears safe to stop mds.cephfs.smithi026.kwjewe 2024-04-16T07:29:57.230 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:56 smithi026 ceph-mon[92215]: pgmap v188: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:57.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:56 smithi104 ceph-mon[81027]: Upgrade: It appears safe to stop mds.cephfs.smithi026.kwjewe 2024-04-16T07:29:57.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:56 smithi104 ceph-mon[81027]: pgmap v188: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:57.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:57.401+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:58.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:58.115+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:58.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:58.402+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: Upgrade: Updating mds.cephfs.smithi026.kwjewe 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:58.830 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:29:58 smithi026 ceph-mon[92215]: Deploying daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: Upgrade: Updating mds.cephfs.smithi026.kwjewe 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.kwjewe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:29:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:29:58 smithi104 ceph-mon[81027]: Deploying daemon mds.cephfs.smithi026.kwjewe on smithi026 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.115+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.116+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.116+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.116+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.116+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:29:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:29:59.116+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:29:59.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:29:59.402+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 202 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa1cfa0 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:00.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.116+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:00.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:00.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:00.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:00.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 258 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f51baa0 2024-04-16T07:30:00.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:00.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.402+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:00.990 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:00 smithi026 ceph-mon[92215]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:00.990 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:00 smithi026 ceph-mon[92215]: overall HEALTH_OK 2024-04-16T07:30:00.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.995+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mdsmap(e 19) v2 ==== 583+0+0 (secure 0 0 0) 0x7fbf8c042070 con 0x7fbf70008390 2024-04-16T07:30:00.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.995+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 19 2024-04-16T07:30:00.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.995+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 msgr2=0x563ccaa1f380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:00.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:00.995+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 0x563ccaa1f380 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.995+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mdsmap(e 19) v2 ==== 583+0+0 (secure 0 0 0) 0x7fe9d002b070 con 0x55fc3f5009a0 2024-04-16T07:30:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.995+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 19 2024-04-16T07:30:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.995+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 msgr2=0x55fc3f51de80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:00.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:00.995+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 0x55fc3f51de80 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mdsmap(e 20) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c02a050 con 0x7fbf70008390 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 20 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 1 client.14524 mds incarnation changed from 0 to 20 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 msgr2=0x563ccaa1f380 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x563ccaa1cfa0 0x563ccaa1f380 unknown :-1 s=CLOSED pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:01.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.007+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mdsmap(e 20) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d001c070 con 0x55fc3f5009a0 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 20 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 1 client.24339 mds incarnation changed from 0 to 20 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 msgr2=0x55fc3f51de80 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6828/1284848687,v1:172.21.15.26:6829/1284848687] conn(0x55fc3f51baa0 0x55fc3f51de80 unknown :-1 s=CLOSED pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.008+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:01.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104418 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 522, success, outsize: 120 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.016+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.016+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.016+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.016+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:01.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.016+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:01.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122200 2024-04-16T07:30:01.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.017+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:01.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.017+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:01.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 410, success, outsize: 120 2024-04-16T07:30:01.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.017+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:01.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.017+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:01.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.018+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:01.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:00 smithi104 ceph-mon[81027]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:01.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:00 smithi104 ceph-mon[81027]: overall HEALTH_OK 2024-04-16T07:30:01.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:01.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:01.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:01.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:01.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:01.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:01.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:01.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:00 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[92211]: 2024-04-16T07:30:00.990+0000 7fbd50600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:01.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.402+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:01.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:01.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.402+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:01.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.403+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:01.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.403+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:01.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:01.403+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T07:30:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: Standby daemon mds.cephfs.smithi104.dpfdje assigned to filesystem cephfs as rank 0 2024-04-16T07:30:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:02.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:02.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:01 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:replay} 2 up:standby 2024-04-16T07:30:02.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mdsmap(e 21) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d000f3a0 con 0x55fc3f5009a0 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 21 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 0x7fe9b4010850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 10 client.24339 send_reconnect to mds.0 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 10 client.24339 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 10 client.24339 path #0x1 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 10 client.24339 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:02.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.018+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fe9c0051580 con 0x55fc3f500090 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mdsmap(e 21) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c024bb0 con 0x7fbf70008390 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 21 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 0x7fbf7000b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 10 client.14524 send_reconnect to mds.0 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 10 client.14524 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 10 client.14524 path #0x1 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 10 client.14524 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:02.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.018+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fbf84016810 con 0x563ccaa01090 2024-04-16T07:30:02.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.019+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 0x7fbf7000b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:02.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.019+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 0x7fe9b4010850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:02.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.021+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.104:6824/815493281 2024-04-16T07:30:02.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.021+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 0x7fe9b4010850 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6283b36c5ee3ce83 server_cookie=db408c05e852fd1f in_seq=0 out_seq=0 2024-04-16T07:30:02.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.021+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6824/815493281 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x7fe9c804c070 con 0x55fc3f500090 2024-04-16T07:30:02.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.021+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:30:02.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.021+0000 7fe9baa00640 10 client.24339 mds.0 already opened, ignore it 2024-04-16T07:30:02.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.022+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.104:6824/815493281 2024-04-16T07:30:02.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.022+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 0x7fbf7000b590 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=278b94e7db90061 server_cookie=87319baa77019501 in_seq=0 out_seq=0 2024-04-16T07:30:02.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.022+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6824/815493281 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa01090 2024-04-16T07:30:02.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.022+0000 7fbf77400640 10 client.14524 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:30:02.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.022+0000 7fbf77400640 10 client.14524 mds.0 already opened, ignore it 2024-04-16T07:30:02.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122201 2024-04-16T07:30:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104431 2024-04-16T07:30:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 524, success, outsize: 120 2024-04-16T07:30:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.026+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:02.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.026+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.026+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.026+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.026+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.026+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 412, success, outsize: 120 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.026+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.026+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.026+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:02.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.026+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: Standby daemon mds.cephfs.smithi104.dpfdje assigned to filesystem cephfs as rank 0 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:01 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:replay} 2 up:standby 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:02.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:02.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:02.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:02.403+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:03.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.031+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 9 ==== mdsmap(e 22) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d000f980 con 0x55fc3f5009a0 2024-04-16T07:30:03.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.031+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 22 2024-04-16T07:30:03.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.030+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 9 ==== mdsmap(e 22) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c024e10 con 0x7fbf70008390 2024-04-16T07:30:03.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.030+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 22 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6824/815493281 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fe9c0051580 con 0x55fc3f500090 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 mds.0 seq now 1 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 update_snap_trace len 70 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0, nref was 0 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0x7fe9b4010fc0, nref now is 2 2024-04-16T07:30:03.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=0 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) seq 1 > 0 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 invalidate_snaprealm_and_children snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 15 client.24339 update_snap_trace snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) self|parent updated 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 15 client.24339 snapc 1=[] 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 10 client.24339 no new snap on snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x3 0x7fe9b4010fc0 3 -> 2 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.038+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x3 0x7fe9b4010fc0 2 -> 1 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6824/815493281 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fbf84016810 con 0x563ccaa01090 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 mds.0 seq now 1 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0, nref was 0 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0x7fbf7000bcb0, nref now is 2 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=0 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) seq 1 > 0 2024-04-16T07:30:03.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 invalidate_snaprealm_and_children snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) 2024-04-16T07:30:03.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 15 client.14524 update_snap_trace snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=0=[] last_modified=0.000000 change_attr=0) self|parent updated 2024-04-16T07:30:03.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 15 client.14524 snapc 1=[] 2024-04-16T07:30:03.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 10 client.14524 no new snap on snaprealm(0x3 nref=3 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) 2024-04-16T07:30:03.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 20 client.14524 put_snap_realm 0x3 0x7fbf7000bcb0 3 -> 2 2024-04-16T07:30:03.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.038+0000 7fbf77400640 20 client.14524 put_snap_realm 0x3 0x7fbf7000bcb0 2 -> 1 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:03.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:03.116+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:03 smithi104 ceph-mon[81027]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:03 smithi104 ceph-mon[81027]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00299996 2024-04-16T07:30:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:03 smithi104 ceph-mon[81027]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00399995 2024-04-16T07:30:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:03 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:reconnect 2024-04-16T07:30:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:03 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:reconnect} 2 up:standby 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:03.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:03.403+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:03 smithi026 ceph-mon[92215]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:03 smithi026 ceph-mon[92215]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00299996 2024-04-16T07:30:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:03 smithi026 ceph-mon[92215]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00399995 2024-04-16T07:30:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:03 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:reconnect 2024-04-16T07:30:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:03 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:reconnect} 2 up:standby 2024-04-16T07:30:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.041+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 10 ==== mdsmap(e 23) v2 ==== 1142+0+0 (secure 0 0 0) 0x7fbf8c0246d0 con 0x7fbf70008390 2024-04-16T07:30:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.041+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 23 2024-04-16T07:30:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.041+0000 7fbf77400640 10 client.14524 kick_requests for mds.0 2024-04-16T07:30:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.041+0000 7fbf77400640 10 client.14524 kick_flushing_caps mds.0 2024-04-16T07:30:04.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.041+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:04.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.042+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 10 ==== mdsmap(e 23) v2 ==== 1142+0+0 (secure 0 0 0) 0x7fe9d0010740 con 0x55fc3f5009a0 2024-04-16T07:30:04.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.042+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 23 2024-04-16T07:30:04.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.042+0000 7fe9baa00640 10 client.24339 kick_requests for mds.0 2024-04-16T07:30:04.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.042+0000 7fe9baa00640 10 client.24339 kick_flushing_caps mds.0 2024-04-16T07:30:04.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.042+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:04.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:04.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:30:04.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:30:04.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:30:04.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_session(request_renewcaps seq 31) v5 -- 0x7fbf68004560 con 0x563ccaa01090 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.118+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6824/815493281 3 ==== client_session(renewcaps seq 31) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x563ccaa01090 2024-04-16T07:30:04.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:04.118+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 31) v1 from mds.0 2024-04-16T07:30:04.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:04 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:rejoin 2024-04-16T07:30:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:04 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:rejoin} 2 up:standby 2024-04-16T07:30:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:04 smithi104 ceph-mon[81027]: daemon mds.cephfs.smithi104.dpfdje is now active in filesystem cephfs as rank 0 2024-04-16T07:30:04.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:04.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:04.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:04.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004430 con 0x55fc3f500090 2024-04-16T07:30:04.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:04.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:04.403+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:04.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:04 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:rejoin 2024-04-16T07:30:04.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:04 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.dpfdje=up:rejoin} 2 up:standby 2024-04-16T07:30:04.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:04 smithi026 ceph-mon[92215]: daemon mds.cephfs.smithi104.dpfdje is now active in filesystem cephfs as rank 0 2024-04-16T07:30:05.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:05.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:05.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:05.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:05.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:05.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:05.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:active 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi104.dpfdje=up:active} 2 up:standby 2024-04-16T07:30:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:05.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:05.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:05.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:05.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004a10 con 0x55fc3f500090 2024-04-16T07:30:05.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:05.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:05.403+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 192 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] up:active 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi104.dpfdje=up:active} 2 up:standby 2024-04-16T07:30:05.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:06.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:06.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.403+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.404+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac004e00 con 0x55fc3f500090 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.404+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:06.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:06.404+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:07.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:07.117+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:07.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:07 smithi026 ceph-mon[92215]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-04-16T07:30:07.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:07 smithi104 ceph-mon[81027]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac005150 con 0x55fc3f500090 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:07.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:07.404+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.117+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.117+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.117+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.117+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.118+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:08.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:08.118+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:08.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] up:boot 2024-04-16T07:30:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi104.dpfdje=up:active} 3 up:standby 2024-04-16T07:30:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:30:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0054a0 con 0x55fc3f500090 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:08.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:08.404+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:08.494 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] up:boot 2024-04-16T07:30:08.494 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi104.dpfdje=up:active} 3 up:standby 2024-04-16T07:30:08.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:30:08.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:08.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:08.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:09.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:09.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:09.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:09.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:09.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:09.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:09.118+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:09.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:09 smithi104 ceph-mon[81027]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.3 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T07:30:09.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:09.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:09.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:09.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0057f0 con 0x55fc3f500090 2024-04-16T07:30:09.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:09.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:09.405+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:09.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:09 smithi026 ceph-mon[92215]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.3 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:10.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:10.118+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:10.388 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:10 smithi026 ceph-mon[92215]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T07:30:10.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:10.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:10.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:10.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac005b90 con 0x55fc3f500090 2024-04-16T07:30:10.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:10.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:10.405+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:10.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:10 smithi104 ceph-mon[81027]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T07:30:11.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:11.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:11.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:11.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:11.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:11.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:11.119+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:11.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:11.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:11.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:11.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac005f30 con 0x55fc3f500090 2024-04-16T07:30:11.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:11.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:11.405+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:12.118+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:12.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:12.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:30:12.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_session(request_renewcaps seq 31) v5 -- 0x7fe9ac005f30 con 0x55fc3f500090 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.405+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.406+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6824/815493281 3 ==== client_session(renewcaps seq 31) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c804c040 con 0x55fc3f500090 2024-04-16T07:30:12.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:12.406+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 31) v1 from mds.0 2024-04-16T07:30:12.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:12 smithi026 ceph-mon[92215]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.4 KiB/s rd, 197 B/s wr, 5 op/s 2024-04-16T07:30:12.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:12 smithi104 ceph-mon[81027]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.4 KiB/s rd, 197 B/s wr, 5 op/s 2024-04-16T07:30:13.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:13.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:13.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:13.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:13.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:13.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:13.119+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:13.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:13.406+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:14.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:14.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:14.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:14.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:14.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:14.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:14.119+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:14.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:14.406+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:14 smithi026 ceph-mon[92215]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:14.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:14 smithi104 ceph-mon[81027]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:14.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:15.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:15.119+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:15.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:15.406+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:16.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:16.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:16.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:16.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:16.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:16.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:16.119+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:16.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:16.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:16.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:16.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:16.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:16.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:16.406+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:16.557 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:16 smithi026 ceph-mon[92215]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:16.557 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:16.557 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:16 smithi104 ceph-mon[81027]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:16.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:17.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.119+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:17.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.119+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:17.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.119+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:17.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.119+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:17.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.120+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:17.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:17.120+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:17.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:17.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:17.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:17.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:17.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:17.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:17.406+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:18.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:18.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:18.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:18.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:18.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:18.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:18.120+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:18.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.406+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:18.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:18.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:18.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.407+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:18.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.407+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:18.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:18.407+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:18.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:18 smithi104 ceph-mon[81027]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-16T07:30:19.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:18 smithi026 ceph-mon[92215]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-16T07:30:19.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:19.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:19.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:19.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 204 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:19.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:19.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.120+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:19.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:19.407+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:19.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123425 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 414, success, outsize: 120 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.461+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.461+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.461+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.461+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.461+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123518 2024-04-16T07:30:19.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 416, success, outsize: 120 2024-04-16T07:30:19.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.492+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.492+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.492+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.492+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.492+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123614 2024-04-16T07:30:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 418, success, outsize: 120 2024-04-16T07:30:19.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.529+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.529+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.529+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.529+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.529+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123716 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 420, success, outsize: 120 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.558+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.558+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.558+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.558+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.558+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123815 2024-04-16T07:30:19.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 422, success, outsize: 120 2024-04-16T07:30:19.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.586+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.586+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.586+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.586+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.586+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123908 2024-04-16T07:30:19.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 424, success, outsize: 120 2024-04-16T07:30:19.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.615+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.615+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.615+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.615+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.615+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124004 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.644+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 426, success, outsize: 120 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.644+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.644+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.644+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.644+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124106 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 428, success, outsize: 120 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.678+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.678+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.678+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.678+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.678+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124205 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 430, success, outsize: 120 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.708+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.708+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.708+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.708+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.708+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124298 2024-04-16T07:30:19.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 432, success, outsize: 120 2024-04-16T07:30:19.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.738+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.738+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.738+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.738+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.738+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124394 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 434, success, outsize: 120 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.765+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.765+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.765+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.765+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.765+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124496 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 436, success, outsize: 120 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.794+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.794+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.794+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.794+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.794+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124595 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 438, success, outsize: 120 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.823+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.823+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.823+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.823+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.823+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124688 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 440, success, outsize: 120 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.852+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.852+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.852+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.852+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.852+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124784 2024-04-16T07:30:19.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 442, success, outsize: 120 2024-04-16T07:30:19.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.881+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.881+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.881+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.881+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.881+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124886 2024-04-16T07:30:19.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 444, success, outsize: 120 2024-04-16T07:30:19.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.910+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.910+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.910+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.910+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.910+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124985 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 446, success, outsize: 120 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.940+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.940+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.940+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.940+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.940+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125078 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 448, success, outsize: 120 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.968+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.968+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.968+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.968+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:19.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:19.968+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:19.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125174 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 450, success, outsize: 120 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.001+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.001+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.001+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.001+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.001+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125276 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 452, success, outsize: 120 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.037+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.037+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.037+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.037+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.037+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125375 2024-04-16T07:30:20.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 454, success, outsize: 120 2024-04-16T07:30:20.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.074+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.074+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.074+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.074+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.074+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125468 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 456, success, outsize: 120 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.111+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.111+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.111+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.111+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.111+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 226 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:20.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.120+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125564 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 458, success, outsize: 120 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.138+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.138+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.138+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.138+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.138+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125666 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 460, success, outsize: 120 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.170+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.170+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.170+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.170+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.170+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125765 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 462, success, outsize: 120 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.201+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.201+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.201+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.201+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.201+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125858 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 464, success, outsize: 120 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.228+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.228+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.228+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.228+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.228+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125954 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 466, success, outsize: 120 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.258+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.258+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.258+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.258+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.258+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126056 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 468, success, outsize: 120 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.288+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.288+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.288+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.288+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:20.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:20.288+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:20.407+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:20.660 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:20 smithi026 ceph-mon[92215]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:30:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:20 smithi104 ceph-mon[81027]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:21.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:21.120+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:21.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:21.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:21.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:21.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:21.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:21.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:21.407+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:22.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:22.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:22.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:22.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:22.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:22.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:22.121+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:22.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 -- 172.21.15.26:0/2041917664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 msgr2=0x7fc314103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 --2- 172.21.15.26:0/2041917664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314103830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc308004940 tx=0x7fc308009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:22.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 -- 172.21.15.26:0/2041917664 shutdown_connections 2024-04-16T07:30:22.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 --2- 172.21.15.26:0/2041917664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 0x7fc314104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 --2- 172.21.15.26:0/2041917664 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314103830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 -- 172.21.15.26:0/2041917664 >> 172.21.15.26:0/2041917664 conn(0x7fc314072640 msgr2=0x7fc314074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:22.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 -- 172.21.15.26:0/2041917664 shutdown_connections 2024-04-16T07:30:22.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.159+0000 7fc31be00700 1 -- 172.21.15.26:0/2041917664 wait complete. 2024-04-16T07:30:22.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 Processor -- start 2024-04-16T07:30:22.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 -- start start 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 0x7fc314114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314114a40 con 0x7fc314103420 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.160+0000 7fc31be00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314114bb0 con 0x7fc3141046d0 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47756/0 (socket says 172.21.15.26:47756) 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 -- 172.21.15.26:0/3965106039 learned_addr learned my addr 172.21.15.26:0/3965106039 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc318a00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 0x7fc314114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 -- 172.21.15.26:0/3965106039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 msgr2=0x7fc314114440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 0x7fc314114440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.161+0000 7fc319400700 1 -- 172.21.15.26:0/3965106039 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308005040 con 0x7fc314103420 2024-04-16T07:30:22.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc319400700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc308013870 tx=0x7fc308009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc308021070 con 0x7fc314103420 2024-04-16T07:30:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc308009430 con 0x7fc314103420 2024-04-16T07:30:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc30800e070 con 0x7fc314103420 2024-04-16T07:30:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3141195b0 con 0x7fc314103420 2024-04-16T07:30:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.162+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc314119af0 con 0x7fc314103420 2024-04-16T07:30:22.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.164+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31404c710 con 0x7fc314103420 2024-04-16T07:30:22.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.165+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc30801c750 con 0x7fc314103420 2024-04-16T07:30:22.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.165+0000 7fc311600700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 0x7fc2fc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.166+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc30808f990 con 0x7fc314103420 2024-04-16T07:30:22.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.166+0000 7fc318a00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 0x7fc2fc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.167+0000 7fc318a00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 0x7fc2fc0710b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc300009b80 tx=0x7fc30000b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:22.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.172+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc30805c330 con 0x7fc314103420 2024-04-16T07:30:22.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.407+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:22.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:22.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:22.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.408+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:22.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.408+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:22.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:22.408+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:22.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.426+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3141089a0 con 0x7fc2fc06ec00 2024-04-16T07:30:22.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi104.dpfdje"]}]: dispatch 2024-04-16T07:30:22.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:22 smithi026 ceph-mon[92215]: Upgrade: It appears safe to stop mds.cephfs.smithi104.dpfdje 2024-04-16T07:30:22.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.429+0000 7fc311600700 1 -- 172.21.15.26:0/3965106039 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc3141089a0 con 0x7fc2fc06ec00 2024-04-16T07:30:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 msgr2=0x7fc2fc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 0x7fc2fc0710b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc300009b80 tx=0x7fc30000b040 comp rx=0 tx=0).stop 2024-04-16T07:30:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 msgr2=0x7fc314113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc308013870 tx=0x7fc308009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 shutdown_connections 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc2fc06ec00 0x7fc2fc0710b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc314103420 0x7fc314113f00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 --2- 172.21.15.26:0/3965106039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3141046d0 0x7fc314114440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 >> 172.21.15.26:0/3965106039 conn(0x7fc314072640 msgr2=0x7fc314073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 shutdown_connections 2024-04-16T07:30:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.432+0000 7fc31be00700 1 -- 172.21.15.26:0/3965106039 wait complete. 2024-04-16T07:30:22.448 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:22.461 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi104.dpfdje"]}]: dispatch 2024-04-16T07:30:22.462 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:22 smithi104 ceph-mon[81027]: Upgrade: It appears safe to stop mds.cephfs.smithi104.dpfdje 2024-04-16T07:30:22.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 -- 172.21.15.26:0/1742762520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 msgr2=0x7fda0c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 --2- 172.21.15.26:0/1742762520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c104b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8004940 tx=0x7fd9f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 -- 172.21.15.26:0/1742762520 shutdown_connections 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 --2- 172.21.15.26:0/1742762520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c104b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 --2- 172.21.15.26:0/1742762520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 -- 172.21.15.26:0/1742762520 >> 172.21.15.26:0/1742762520 conn(0x7fda0c072640 msgr2=0x7fda0c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 -- 172.21.15.26:0/1742762520 shutdown_connections 2024-04-16T07:30:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.588+0000 7fda10c00700 1 -- 172.21.15.26:0/1742762520 wait complete. 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.589+0000 7fda10c00700 1 Processor -- start 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.589+0000 7fda10c00700 1 -- start start 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda10c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda10c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda10c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c114a90 con 0x7fda0c1046d0 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda10c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c114c00 con 0x7fda0c103420 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47768/0 (socket says 172.21.15.26:47768) 2024-04-16T07:30:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09400700 1 -- 172.21.15.26:0/214637143 learned_addr learned my addr 172.21.15.26:0/214637143 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:22.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09e00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09400700 1 -- 172.21.15.26:0/214637143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 msgr2=0x7fda0c113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.590+0000 7fda09400700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda09400700 1 -- 172.21.15.26:0/214637143 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f8005040 con 0x7fda0c1046d0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda09e00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda09400700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800b3b0 tx=0x7fd9f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f800e040 con 0x7fda0c1046d0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9f8004290 con 0x7fda0c1046d0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f801c630 con 0x7fda0c1046d0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c119600 con 0x7fda0c1046d0 2024-04-16T07:30:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.591+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c119bc0 con 0x7fda0c1046d0 2024-04-16T07:30:22.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.592+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c04c710 con 0x7fda0c1046d0 2024-04-16T07:30:22.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.594+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9f800bbf0 con 0x7fda0c1046d0 2024-04-16T07:30:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.595+0000 7fda02000700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 0x7fd9f00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.595+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd9f808fcf0 con 0x7fda0c1046d0 2024-04-16T07:30:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.595+0000 7fda09e00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 0x7fd9f00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.596+0000 7fda09e00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 0x7fd9f00710b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc007fd0 tx=0x7fd9fc009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:22.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.600+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd9f805c5d0 con 0x7fda0c1046d0 2024-04-16T07:30:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.848+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda0c108a10 con 0x7fd9f006ec00 2024-04-16T07:30:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.851+0000 7fda02000700 1 -- 172.21.15.26:0/214637143 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fda0c108a10 con 0x7fd9f006ec00 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 msgr2=0x7fd9f00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 0x7fd9f00710b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc007fd0 tx=0x7fd9fc009040 comp rx=0 tx=0).stop 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 msgr2=0x7fda0c114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800b3b0 tx=0x7fd9f8009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 shutdown_connections 2024-04-16T07:30:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9f006ec00 0x7fd9f00710b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda0c103420 0x7fda0c113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 --2- 172.21.15.26:0/214637143 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda0c1046d0 0x7fda0c114490 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.853+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 >> 172.21.15.26:0/214637143 conn(0x7fda0c072640 msgr2=0x7fda0c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.854+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 shutdown_connections 2024-04-16T07:30:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:22.854+0000 7fda10c00700 1 -- 172.21.15.26:0/214637143 wait complete. 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 -- 172.21.15.26:0/72665230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 msgr2=0x7fd93c0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 --2- 172.21.15.26:0/72665230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c0703b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd92c004940 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 -- 172.21.15.26:0/72665230 shutdown_connections 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 --2- 172.21.15.26:0/72665230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 0x7fd93c070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 --2- 172.21.15.26:0/72665230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c0703b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 -- 172.21.15.26:0/72665230 >> 172.21.15.26:0/72665230 conn(0x7fd93c0f8390 msgr2=0x7fd93c0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 -- 172.21.15.26:0/72665230 shutdown_connections 2024-04-16T07:30:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.005+0000 7fd940a00700 1 -- 172.21.15.26:0/72665230 wait complete. 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 Processor -- start 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 -- start start 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 0x7fd93c10f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c110520 con 0x7fd93c071ea0 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.006+0000 7fd940a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c110d70 con 0x7fd93c0708f0 2024-04-16T07:30:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47784/0 (socket says 172.21.15.26:47784) 2024-04-16T07:30:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 -- 172.21.15.26:0/4279707843 learned_addr learned my addr 172.21.15.26:0/4279707843 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93b400700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 0x7fd93c10f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 -- 172.21.15.26:0/4279707843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 msgr2=0x7fd93c10f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 0x7fd93c10f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.007+0000 7fd93aa00700 1 -- 172.21.15.26:0/4279707843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c005040 con 0x7fd93c071ea0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd93aa00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd92800eb40 tx=0x7fd92800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd928007cc0 con 0x7fd93c071ea0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd92800f040 con 0x7fd93c071ea0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c110ff0 con 0x7fd93c071ea0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd92800ca10 con 0x7fd93c071ea0 2024-04-16T07:30:23.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.008+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c111530 con 0x7fd93c071ea0 2024-04-16T07:30:23.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.009+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c0fc030 con 0x7fd93c071ea0 2024-04-16T07:30:23.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.011+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd92800cb70 con 0x7fd93c071ea0 2024-04-16T07:30:23.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.012+0000 7fd933400700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 0x7fd9200710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.012+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd92808e470 con 0x7fd93c071ea0 2024-04-16T07:30:23.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.012+0000 7fd93b400700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 0x7fd9200710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.013+0000 7fd93b400700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 0x7fd9200710b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd92c013870 tx=0x7fd92c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.017+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd92805ad80 con 0x7fd93c071ea0 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:23.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:23.121+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:23.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.260+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd93c111920 con 0x7fd92006ec00 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.289+0000 7fd933400700 1 -- 172.21.15.26:0/4279707843 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd93c111920 con 0x7fd92006ec00 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (7m) 9s ago 17m 14.0M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (4m) 9s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (4m) 63s ago 16m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:30:23.289 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (7m) 9s ago 17m 73.0M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (44s) 9s ago 10m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (16s) 9s ago 10m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (10m) 63s ago 10m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 0ae35be12158 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (10m) 63s ago 10m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (6m) 9s ago 19m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (6m) 63s ago 16m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (5m) 9s ago 19m 49.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (5m) 63s ago 16m 36.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (7m) 9s ago 17m 8061k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 63s ago 16m 8070k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (3m) 9s ago 15m 51.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (3m) 9s ago 14m 51.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:30:23.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (2m) 9s ago 14m 43.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (2m) 63s ago 13m 50.9M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (96s) 63s ago 13m 41.4M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (68s) 63s ago 12m 11.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (6m) 9s ago 17m 60.1M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 msgr2=0x7fd9200710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 0x7fd9200710b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd92c013870 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:23.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 msgr2=0x7fd93c10ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd92800eb40 tx=0x7fd92800b040 comp rx=0 tx=0).stop 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 shutdown_connections 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd92006ec00 0x7fd9200710b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd93c0708f0 0x7fd93c10f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 --2- 172.21.15.26:0/4279707843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd93c071ea0 0x7fd93c10ff20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 >> 172.21.15.26:0/4279707843 conn(0x7fd93c0f8390 msgr2=0x7fd93c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:23.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.292+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 shutdown_connections 2024-04-16T07:30:23.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.293+0000 7fd940a00700 1 -- 172.21.15.26:0/4279707843 wait complete. 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:23.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:23.408+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.451+0000 7ff04f000700 1 -- 172.21.15.26:0/689205629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0481002d0 msgr2=0x7ff048100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.451+0000 7ff04f000700 1 --2- 172.21.15.26:0/689205629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0481002d0 0x7ff048100740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff034004940 tx=0x7ff034009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 -- 172.21.15.26:0/689205629 shutdown_connections 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 --2- 172.21.15.26:0/689205629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0481002d0 0x7ff048100740 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 --2- 172.21.15.26:0/689205629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0480ff020 0x7ff0480ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 -- 172.21.15.26:0/689205629 >> 172.21.15.26:0/689205629 conn(0x7ff0480fa950 msgr2=0x7ff0480fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:23.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:23 smithi026 ceph-mon[92215]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:23 smithi026 ceph-mon[92215]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:23.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 -- 172.21.15.26:0/689205629 shutdown_connections 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.452+0000 7ff04f000700 1 -- 172.21.15.26:0/689205629 wait complete. 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 Processor -- start 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 -- start start 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0481002d0 0x7ff0480761c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048072720 con 0x7ff0480ff020 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff04f000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048072890 con 0x7ff0481002d0 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.453+0000 7ff047e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47796/0 (socket says 172.21.15.26:47796) 2024-04-16T07:30:23.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 -- 172.21.15.26:0/941626880 learned_addr learned my addr 172.21.15.26:0/941626880 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047400700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0481002d0 0x7ff0480761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 -- 172.21.15.26:0/941626880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0481002d0 msgr2=0x7ff0480761c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0481002d0 0x7ff0480761c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 -- 172.21.15.26:0/941626880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff034005040 con 0x7ff0480ff020 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.454+0000 7ff047e00700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c830 tx=0x7ff038004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.455+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff03800a760 con 0x7ff0480ff020 2024-04-16T07:30:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.455+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff03800ad90 con 0x7ff0480ff020 2024-04-16T07:30:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.455+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff048072b20 con 0x7ff0480ff020 2024-04-16T07:30:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.455+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0380113a0 con 0x7ff0480ff020 2024-04-16T07:30:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.455+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0480730e0 con 0x7ff0480ff020 2024-04-16T07:30:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.456+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff048108320 con 0x7ff0480ff020 2024-04-16T07:30:23.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.457+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff038012400 con 0x7ff0480ff020 2024-04-16T07:30:23.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.458+0000 7ff03fe00700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 0x7ff02c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.459+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff03808e910 con 0x7ff0480ff020 2024-04-16T07:30:23.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.459+0000 7ff047400700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 0x7ff02c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.460+0000 7ff047400700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 0x7ff02c0710b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff03400b3b0 tx=0x7ff034009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.464+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff03805b1f0 con 0x7ff0480ff020 2024-04-16T07:30:23.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:23 smithi104 ceph-mon[81027]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:23.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:23 smithi104 ceph-mon[81027]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:23.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.782+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff04804c710 con 0x7ff0480ff020 2024-04-16T07:30:23.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.783+0000 7ff03fe00700 1 -- 172.21.15.26:0/941626880 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ff03805ad80 con 0x7ff0480ff020 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T07:30:23.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 12 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:30:23.784 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 msgr2=0x7ff02c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 0x7ff02c0710b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff03400b3b0 tx=0x7ff034009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 msgr2=0x7ff048075c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c830 tx=0x7ff038004470 comp rx=0 tx=0).stop 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 shutdown_connections 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff02c06ec00 0x7ff02c0710b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0480ff020 0x7ff048075c80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 --2- 172.21.15.26:0/941626880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0481002d0 0x7ff0480761c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.786+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 >> 172.21.15.26:0/941626880 conn(0x7ff0480fa950 msgr2=0x7ff0480fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.787+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 shutdown_connections 2024-04-16T07:30:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.787+0000 7ff04f000700 1 -- 172.21.15.26:0/941626880 wait complete. 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 -- 172.21.15.26:0/3237137887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 msgr2=0x7fe690100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 --2- 172.21.15.26:0/3237137887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe690100740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004940 tx=0x7fe67c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 -- 172.21.15.26:0/3237137887 shutdown_connections 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 --2- 172.21.15.26:0/3237137887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe690100740 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 --2- 172.21.15.26:0/3237137887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 0x7fe6900ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 -- 172.21.15.26:0/3237137887 >> 172.21.15.26:0/3237137887 conn(0x7fe6900fa950 msgr2=0x7fe6900fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:23.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 -- 172.21.15.26:0/3237137887 shutdown_connections 2024-04-16T07:30:23.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.930+0000 7fe695400700 1 -- 172.21.15.26:0/3237137887 wait complete. 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 Processor -- start 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 -- start start 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 0x7fe69010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901106b0 con 0x7fe6901002d0 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe695400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690110820 con 0x7fe6900ff020 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe68de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe68de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47812/0 (socket says 172.21.15.26:47812) 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.931+0000 7fe68de00700 1 -- 172.21.15.26:0/3784519527 learned_addr learned my addr 172.21.15.26:0/3784519527 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe68e800700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 0x7fe69010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe68de00700 1 -- 172.21.15.26:0/3784519527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 msgr2=0x7fe69010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:23.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe68de00700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 0x7fe69010fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:23.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe68de00700 1 -- 172.21.15.26:0/3784519527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c005040 con 0x7fe6901002d0 2024-04-16T07:30:23.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe68de00700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009150 tx=0x7fe67c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.932+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe67c00e040 con 0x7fe6901002d0 2024-04-16T07:30:23.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.933+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe67c00bbf0 con 0x7fe6901002d0 2024-04-16T07:30:23.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.933+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe690115220 con 0x7fe6901002d0 2024-04-16T07:30:23.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.933+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe67c01c630 con 0x7fe6901002d0 2024-04-16T07:30:23.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.933+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901157e0 con 0x7fe6901002d0 2024-04-16T07:30:23.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.934+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69004c710 con 0x7fe6901002d0 2024-04-16T07:30:23.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.935+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe67c00bd60 con 0x7fe6901002d0 2024-04-16T07:30:23.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.936+0000 7fe686a00700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 0x7fe6740710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:23.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.936+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe67c08fc90 con 0x7fe6901002d0 2024-04-16T07:30:23.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.937+0000 7fe68e800700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 0x7fe6740710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:23.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.937+0000 7fe68e800700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 0x7fe6740710b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe680004990 tx=0x7fe680009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:23.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:23.941+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe67c05c570 con 0x7fe6901002d0 2024-04-16T07:30:24.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:24.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_session(request_renewcaps seq 32) v5 -- 0x7fbf68004560 con 0x563ccaa01090 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.121+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.122+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6824/815493281 4 ==== client_session(renewcaps seq 32) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf84009b00 con 0x563ccaa01090 2024-04-16T07:30:24.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:24.122+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 32) v1 from mds.0 2024-04-16T07:30:24.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.215+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe690107b10 con 0x7fe6901002d0 2024-04-16T07:30:24.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.216+0000 7fe686a00700 1 -- 172.21.15.26:0/3784519527 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1893 (secure 0 0 0) 0x7fe67c05c100 con 0x7fe6901002d0 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:e24 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 23 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:30:24.216 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:04.031949+0000 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 83 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=24287} 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:30:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 24287 members: 24287 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{0:24287} state up:active seq 157 join_fscid=1 addr [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:24303} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T07:30:24.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{-1:34404} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 msgr2=0x7fe6740710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 0x7fe6740710b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe680004990 tx=0x7fe680009ee0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 msgr2=0x7fe6901100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009150 tx=0x7fe67c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:24.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 shutdown_connections 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe67406ec00 0x7fe6740710b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6900ff020 0x7fe69010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 --2- 172.21.15.26:0/3784519527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6901002d0 0x7fe6901100b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.219+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 >> 172.21.15.26:0/3784519527 conn(0x7fe6900fa950 msgr2=0x7fe6900fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.220+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 shutdown_connections 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.220+0000 7fe695400700 1 -- 172.21.15.26:0/3784519527 wait complete. 2024-04-16T07:30:24.220 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2024-04-16T07:30:24.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3387833668 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 msgr2=0x7f4f88102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3387833668 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88102340 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f74004940 tx=0x7f4f74009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3387833668 shutdown_connections 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3387833668 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88102340 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3387833668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 0x7f4f880ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3387833668 >> 172.21.15.26:0/3387833668 conn(0x7f4f880f7450 msgr2=0x7f4f880f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3387833668 shutdown_connections 2024-04-16T07:30:24.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.383+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3387833668 wait complete. 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.384+0000 7f4f8ea00700 1 Processor -- start 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.384+0000 7f4f8ea00700 1 -- start start 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f8ea00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 0x7f4f8810fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f8ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f8ea00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88110870 con 0x7f4f880fff20 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f8ea00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88110fb0 con 0x7f4f88066b80 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47834/0 (socket says 172.21.15.26:47834) 2024-04-16T07:30:24.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 -- 172.21.15.26:0/3832450232 learned_addr learned my addr 172.21.15.26:0/3832450232 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:24.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87e00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 0x7f4f8810fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 -- 172.21.15.26:0/3832450232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 msgr2=0x7f4f8810fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 0x7f4f8810fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.385+0000 7f4f87400700 1 -- 172.21.15.26:0/3832450232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74005040 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f87400700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400b3b0 tx=0x7f4f74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7400e040 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f74004290 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7401c630 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f88111230 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.386+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f881117f0 con 0x7f4f880fff20 2024-04-16T07:30:24.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.387+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f880f8fd0 con 0x7f4f880fff20 2024-04-16T07:30:24.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.389+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f7400bbf0 con 0x7f4f880fff20 2024-04-16T07:30:24.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.389+0000 7f4f7fe00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 0x7f4f6c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.390+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f4f7408fd00 con 0x7f4f880fff20 2024-04-16T07:30:24.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.390+0000 7f4f87e00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 0x7f4f6c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.391+0000 7f4f87e00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 0x7f4f6c0710c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f880668c0 tx=0x7f4f78009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:24.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.395+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4f74093030 con 0x7f4f880fff20 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:24.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:24.408+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/941626880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: Upgrade: Updating mds.cephfs.smithi104.dpfdje 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:24.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:24 smithi026 ceph-mon[92215]: Deploying daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/941626880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: Upgrade: Updating mds.cephfs.smithi104.dpfdje 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.dpfdje", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:24.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:24 smithi104 ceph-mon[81027]: Deploying daemon mds.cephfs.smithi104.dpfdje on smithi104 2024-04-16T07:30:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.649+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f88105ab0 con 0x7f4f6c06ec10 2024-04-16T07:30:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.652+0000 7f4f7fe00700 1 -- 172.21.15.26:0/3832450232 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4f88105ab0 con 0x7f4f6c06ec10 2024-04-16T07:30:24.651 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "14/21 daemons upgraded", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:30:24.652 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:30:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 msgr2=0x7f4f6c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 0x7f4f6c0710c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f880668c0 tx=0x7f4f78009ee0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 msgr2=0x7f4f88110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400b3b0 tx=0x7f4f74009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 shutdown_connections 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4f6c06ec10 0x7f4f6c0710c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f88066b80 0x7f4f8810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 --2- 172.21.15.26:0/3832450232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f880fff20 0x7f4f88110270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 >> 172.21.15.26:0/3832450232 conn(0x7f4f880f7450 msgr2=0x7f4f880fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 shutdown_connections 2024-04-16T07:30:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.655+0000 7f4f8ea00700 1 -- 172.21.15.26:0/3832450232 wait complete. 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.816+0000 7faecde00700 1 -- 172.21.15.26:0/1925754697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fe4d0 msgr2=0x7faec80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.816+0000 7faecde00700 1 --2- 172.21.15.26:0/1925754697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fe4d0 0x7faec80fe8e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004940 tx=0x7faeb8009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 -- 172.21.15.26:0/1925754697 shutdown_connections 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 --2- 172.21.15.26:0/1925754697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fee20 0x7faec80faf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 --2- 172.21.15.26:0/1925754697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fe4d0 0x7faec80fe8e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 -- 172.21.15.26:0/1925754697 >> 172.21.15.26:0/1925754697 conn(0x7faec80f6bd0 msgr2=0x7faec80f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 -- 172.21.15.26:0/1925754697 shutdown_connections 2024-04-16T07:30:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.817+0000 7faecde00700 1 -- 172.21.15.26:0/1925754697 wait complete. 2024-04-16T07:30:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 Processor -- start 2024-04-16T07:30:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 -- start start 2024-04-16T07:30:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fe4d0 0x7faec80fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec80fc740 con 0x7faec80fee20 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.818+0000 7faecde00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec80fc8b0 con 0x7faec80fe4d0 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47864/0 (socket says 172.21.15.26:47864) 2024-04-16T07:30:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 -- 172.21.15.26:0/2415550479 learned_addr learned my addr 172.21.15.26:0/2415550479 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6a00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fe4d0 0x7faec80fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 -- 172.21.15.26:0/2415550479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fe4d0 msgr2=0x7faec80fdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fe4d0 0x7faec80fdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 -- 172.21.15.26:0/2415550479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8005040 con 0x7faec80fee20 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.819+0000 7faec6000700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faeb4002a50 tx=0x7faeb40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.820+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb4007cc0 con 0x7faec80fee20 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.820+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faeb400f040 con 0x7faec80fee20 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.820+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb4009640 con 0x7faec80fee20 2024-04-16T07:30:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.820+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec80fcb90 con 0x7faec80fee20 2024-04-16T07:30:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.820+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec8115a50 con 0x7faec80fee20 2024-04-16T07:30:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.821+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec805f5f0 con 0x7faec80fee20 2024-04-16T07:30:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.822+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faeb40097a0 con 0x7faec80fee20 2024-04-16T07:30:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.823+0000 7faebea00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 0x7faeac0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.823+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6582+0+0 (secure 0 0 0) 0x7faeb408e6b0 con 0x7faec80fee20 2024-04-16T07:30:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.823+0000 7faec6a00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 0x7faeac0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.824+0000 7faec6a00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 0x7faeac0710c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faeb800b3b0 tx=0x7faeb8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:24.828+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faeb40573e0 con 0x7faec80fee20 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:25.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:25.121+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.152+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faec804bed0 con 0x7faec80fee20 2024-04-16T07:30:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.153+0000 7faebea00700 1 -- 172.21.15.26:0/2415550479 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7faeb405aaa0 con 0x7faec80fee20 2024-04-16T07:30:25.152 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:30:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.155+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 msgr2=0x7faeac0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.155+0000 7faecde00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 0x7faeac0710c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faeb800b3b0 tx=0x7faeb8009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.155+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 msgr2=0x7faec80fc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.155+0000 7faecde00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faeb4002a50 tx=0x7faeb40046b0 comp rx=0 tx=0).stop 2024-04-16T07:30:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 shutdown_connections 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7faeac06ec10 0x7faeac0710c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec80fe4d0 0x7faec80fdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 --2- 172.21.15.26:0/2415550479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faec80fee20 0x7faec80fc190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 >> 172.21.15.26:0/2415550479 conn(0x7faec80f6bd0 msgr2=0x7faec80f82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 shutdown_connections 2024-04-16T07:30:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:25.156+0000 7faecde00700 1 -- 172.21.15.26:0/2415550479 wait complete. 2024-04-16T07:30:25.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:25.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:25.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:25.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:25.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:25.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:25.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:25.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:25 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3784519527' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:30:25.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:25 smithi026 ceph-mon[92215]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:25.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:25 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2415550479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:30:25.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:25 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3784519527' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:30:25.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:25 smithi104 ceph-mon[81027]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:25.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:25 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2415550479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 232 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x563ccaa01090 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:26.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.122+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:26.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:26 smithi104 ceph-mon[81027]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.408+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.408+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.408+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 260 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x55fc3f500090 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:26.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:26.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:26 smithi026 ceph-mon[92215]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:26.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.917+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 11 ==== mdsmap(e 25) v2 ==== 583+0+0 (secure 0 0 0) 0x7fbf8c019480 con 0x7fbf70008390 2024-04-16T07:30:26.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.917+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 25 2024-04-16T07:30:26.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.917+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 msgr2=0x7fbf7000b590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:26.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.917+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 0x7fbf7000b590 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:26.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.918+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 11 ==== mdsmap(e 25) v2 ==== 583+0+0 (secure 0 0 0) 0x7fe9d00108e0 con 0x55fc3f5009a0 2024-04-16T07:30:26.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.918+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 25 2024-04-16T07:30:26.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.918+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 msgr2=0x7fe9b4010850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:26.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.918+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 0x7fe9b4010850 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 12 ==== mdsmap(e 26) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d0020930 con 0x55fc3f5009a0 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 26 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 1 client.24339 mds incarnation changed from 0 to 26 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 msgr2=0x7fe9b4010850 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x55fc3f500090 0x7fe9b4010850 unknown :-1 s=CLOSED pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.929+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 12 ==== mdsmap(e 26) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c019b00 con 0x7fbf70008390 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 26 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 1 client.14524 mds incarnation changed from 0 to 26 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 msgr2=0x7fbf7000b590 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6824/815493281,v1:172.21.15.104:6825/815493281] conn(0x563ccaa01090 0x7fbf7000b590 unknown :-1 s=CLOSED pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:26.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.928+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:26.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104799 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 526, success, outsize: 120 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.936+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.936+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.936+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.936+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:26.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:26.936+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:26.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126298 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.938+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 470, success, outsize: 120 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.938+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.938+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.938+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:26.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:26.938+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:27.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.122+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:27.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:26 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[92211]: 2024-04-16T07:30:26.913+0000 7fbd50600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:27.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: Standby daemon mds.cephfs.smithi104.zwhiwr assigned to filesystem cephfs as rank 0 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:27.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:27 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:replay} 2 up:standby 2024-04-16T07:30:27.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: Standby daemon mds.cephfs.smithi104.zwhiwr assigned to filesystem cephfs as rank 0 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:27.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:27 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:replay} 2 up:standby 2024-04-16T07:30:27.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 13 ==== mdsmap(e 27) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c019b00 con 0x7fbf70008390 2024-04-16T07:30:27.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 27 2024-04-16T07:30:27.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 0x7fbf7000c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 10 client.14524 send_reconnect to mds.0 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 10 client.14524 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 10 client.14524 path #0x1 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 10 client.14524 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:27.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.931+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fbf84016810 con 0x7fbf7000bfd0 2024-04-16T07:30:27.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.932+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 0x7fbf7000c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:27.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 13 ==== mdsmap(e 27) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d0020930 con 0x55fc3f5009a0 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 27 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 0x7fe9b4013640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 10 client.24339 send_reconnect to mds.0 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 10 client.24339 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 10 client.24339 path #0x1 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 10 client.24339 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.931+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fe9c0051580 con 0x7fe9b4011260 2024-04-16T07:30:27.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.932+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 0x7fe9b4013640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:27.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.934+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 0x7fbf7000c3a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2d3ee14041a03b server_cookie=c9cfb699a07eb45e in_seq=0 out_seq=0 2024-04-16T07:30:27.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.934+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.104:6826/2349406764 2024-04-16T07:30:27.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.934+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6826/2349406764 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000bfd0 2024-04-16T07:30:27.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.935+0000 7fbf77400640 10 client.14524 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.935+0000 7fbf77400640 10 client.14524 mds.0 already opened, ignore it 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.934+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 0x7fe9b4013640 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c9efa8ee02c5bd1 server_cookie=ce7eb88889129542 in_seq=0 out_seq=0 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.934+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.104:6826/2349406764 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.935+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6826/2349406764 1 ==== client_session(open) v5 ==== 66+0+0 (crc 0 0 0) 0x7fe9ac005f30 con 0x7fe9b4011260 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.935+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(open) v5 from mds.0 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.935+0000 7fe9baa00640 10 client.24339 mds.0 already opened, ignore it 2024-04-16T07:30:27.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126302 2024-04-16T07:30:27.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 472, success, outsize: 120 2024-04-16T07:30:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.937+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.937+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.937+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.937+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:27.937+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:27.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104809 2024-04-16T07:30:27.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 528, success, outsize: 120 2024-04-16T07:30:27.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.941+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:27.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.941+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:27.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.941+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:27.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.941+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:27.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:27.941+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:28.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:28.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.122+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:28.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:28.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.935+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 14 ==== mdsmap(e 28) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fe9d0020930 con 0x55fc3f5009a0 2024-04-16T07:30:28.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.935+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 28 2024-04-16T07:30:28.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.935+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 14 ==== mdsmap(e 28) v2 ==== 1134+0+0 (secure 0 0 0) 0x7fbf8c019b00 con 0x7fbf70008390 2024-04-16T07:30:28.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.935+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 28 2024-04-16T07:30:28.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6826/2349406764 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fbf84016810 con 0x7fbf7000bfd0 2024-04-16T07:30:28.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 10 client.14524 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:28.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 10 client.14524 mds.0 seq now 1 2024-04-16T07:30:28.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0x7fbf7000bcb0, nref was 1 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0x7fbf7000bcb0, nref now is 2 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:28.943+0000 7fbf77400640 20 client.14524 put_snap_realm 0x3 0x7fbf7000bcb0 2 -> 1 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6826/2349406764 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fe9c0051580 con 0x7fe9b4011260 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 10 client.24339 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 10 client.24339 mds.0 seq now 1 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 10 client.24339 update_snap_trace len 70 2024-04-16T07:30:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0x7fe9b4010fc0, nref was 1 2024-04-16T07:30:28.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0x7fe9b4010fc0, nref now is 2 2024-04-16T07:30:28.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 10 client.24339 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:30:28.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:28.943+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x3 0x7fe9b4010fc0 2 -> 1 2024-04-16T07:30:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:29.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:29.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:29.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:29.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:29.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.122+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:28 smithi026 ceph-mon[92215]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:28 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:reconnect 2024-04-16T07:30:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:28 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:reconnect} 2 up:standby 2024-04-16T07:30:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:28 smithi026 ceph-mon[92215]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00299996 2024-04-16T07:30:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:28 smithi026 ceph-mon[92215]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00399995 2024-04-16T07:30:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:28 smithi104 ceph-mon[81027]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:28 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:reconnect 2024-04-16T07:30:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:28 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:reconnect} 2 up:standby 2024-04-16T07:30:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:28 smithi104 ceph-mon[81027]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00299996 2024-04-16T07:30:29.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:28 smithi104 ceph-mon[81027]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00399995 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:29.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:29.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.938+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 15 ==== mdsmap(e 29) v2 ==== 1142+0+0 (secure 0 0 0) 0x7fe9d0020940 con 0x55fc3f5009a0 2024-04-16T07:30:29.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.938+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 29 2024-04-16T07:30:29.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.938+0000 7fe9baa00640 10 client.24339 kick_requests for mds.0 2024-04-16T07:30:29.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.938+0000 7fe9baa00640 10 client.24339 kick_flushing_caps mds.0 2024-04-16T07:30:29.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:29.938+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:29.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.938+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 15 ==== mdsmap(e 29) v2 ==== 1142+0+0 (secure 0 0 0) 0x7fbf8c019ce0 con 0x7fbf70008390 2024-04-16T07:30:29.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.938+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 29 2024-04-16T07:30:29.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.938+0000 7fbf77400640 10 client.14524 kick_requests for mds.0 2024-04-16T07:30:29.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.938+0000 7fbf77400640 10 client.14524 kick_flushing_caps mds.0 2024-04-16T07:30:29.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:29.938+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:30.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:30.123+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:rejoin 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:rejoin} 2 up:standby 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: daemon mds.cephfs.smithi104.zwhiwr is now active in filesystem cephfs as rank 0 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:active 2024-04-16T07:30:30.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:29 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi104.zwhiwr=up:active} 2 up:standby 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:rejoin 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi104.zwhiwr=up:rejoin} 2 up:standby 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: daemon mds.cephfs.smithi104.zwhiwr is now active in filesystem cephfs as rank 0 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] up:active 2024-04-16T07:30:30.266 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:29 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi104.zwhiwr=up:active} 2 up:standby 2024-04-16T07:30:30.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:30.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:30.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:30.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac005f30 con 0x7fe9b4011260 2024-04-16T07:30:30.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:30.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:30.409+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:31.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:31.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:31.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:31.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:31.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:31.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:31.123+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:31.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:30 smithi026 ceph-mon[92215]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T07:30:31.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:30 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:30:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:30 smithi104 ceph-mon[81027]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T07:30:31.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:30 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac005f30 con 0x7fe9b4011260 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:31.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:31.410+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:32.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:32.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:32.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:32.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:32.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:32.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:32.123+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_session(request_renewcaps seq 32) v5 -- 0x7fe9ac005f30 con 0x7fe9b4011260 2024-04-16T07:30:32.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.410+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.411+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6826/2349406764 3 ==== client_session(renewcaps seq 32) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c804c040 con 0x7fe9b4011260 2024-04-16T07:30:32.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:32.411+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 32) v1 from mds.0 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:33.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:33.123+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:32 smithi026 ceph-mon[92215]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T07:30:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:32 smithi104 ceph-mon[81027]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T07:30:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:33.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:33.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:33.410+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:34.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:34.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:34.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:34.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:34.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:34.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:34.124+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:34.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:34.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:35.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:35.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:35.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:35.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:35.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:35.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:35.124+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:35.228 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:34 smithi104 ceph-mon[81027]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T07:30:35.228 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:35.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:34 smithi026 ceph-mon[92215]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T07:30:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.410+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:35.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:35.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:36.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:36.124+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:36.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:36.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:37.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:37.125+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:36 smithi026 ceph-mon[92215]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T07:30:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:36 smithi104 ceph-mon[81027]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T07:30:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:37.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:37.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:38 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] up:boot 2024-04-16T07:30:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:38 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi104.zwhiwr=up:active} 3 up:standby 2024-04-16T07:30:38.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:38.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:38.125+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:38 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] up:boot 2024-04-16T07:30:38.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:38 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi104.zwhiwr=up:active} 3 up:standby 2024-04-16T07:30:38.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:30:38.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:38.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:38.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:38.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:38.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:38.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:38.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:39.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:39.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:39.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:39.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:39.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:39.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:39.125+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:39.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:39 smithi026 ceph-mon[92215]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.5 KiB/s rd, 196 B/s wr, 4 op/s 2024-04-16T07:30:39.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:39 smithi104 ceph-mon[81027]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.5 KiB/s rd, 196 B/s wr, 4 op/s 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:39.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:39.411+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:40.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:40.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:40.125+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:40.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:40.412+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:41.126+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:41 smithi026 ceph-mon[92215]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:41 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:41 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:41 smithi104 ceph-mon[81027]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:41 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:41.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:41 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:41.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:41.412+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:42.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:42.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:42.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:42.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:42.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:42.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:42.126+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:42.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:42.412+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:43.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:43.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:43.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:43.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:43.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:43.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:43.126+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:43 smithi104 ceph-mon[81027]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:43 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:43 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:43.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:43.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:43.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:43.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:43.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:43.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:43.412+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:43.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:43 smithi026 ceph-mon[92215]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T07:30:43.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:43 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:43.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:43 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_session(request_renewcaps seq 33) v5 -- 0x7fbf68004560 con 0x7fbf7000bfd0 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:44.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.126+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.127+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.104:6826/2349406764 3 ==== client_session(renewcaps seq 33) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf84009e30 con 0x7fbf7000bfd0 2024-04-16T07:30:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:44.127+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 33) v1 from mds.0 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:44.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:44.413+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:45.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:45.127+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:45.220 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:45 smithi104 ceph-mon[81027]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:45.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:45.413+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:45.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:45 smithi026 ceph-mon[92215]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T07:30:46.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:46.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:46.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:46.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:46.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:46.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:46.127+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.413+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.413+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.413+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 262 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.414+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:46.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.414+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:46.659 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105844 2024-04-16T07:30:46.659 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 530, success, outsize: 120 2024-04-16T07:30:46.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.661+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.661+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.661+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.661+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.661+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105932 2024-04-16T07:30:46.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 532, success, outsize: 120 2024-04-16T07:30:46.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.685+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.685+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.685+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.685+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.685+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106023 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 534, success, outsize: 120 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.711+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.711+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.711+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.711+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.711+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106120 2024-04-16T07:30:46.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.741+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 536, success, outsize: 120 2024-04-16T07:30:46.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.741+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.741+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.741+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.741+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106214 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.769+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 538, success, outsize: 120 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.769+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.769+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.769+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.769+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106302 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 540, success, outsize: 120 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.795+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.795+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.795+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.795+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.795+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106393 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 542, success, outsize: 120 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.813+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.813+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.813+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.813+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.813+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106490 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 544, success, outsize: 120 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.832+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.832+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.832+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.832+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.832+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106584 2024-04-16T07:30:46.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 546, success, outsize: 120 2024-04-16T07:30:46.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.851+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.851+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.851+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.851+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.851+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106672 2024-04-16T07:30:46.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 548, success, outsize: 120 2024-04-16T07:30:46.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.868+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.868+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.868+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.868+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.868+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106763 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 550, success, outsize: 120 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.888+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.888+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.888+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.888+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.888+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106860 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 552, success, outsize: 120 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.910+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.910+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.910+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.910+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.910+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106954 2024-04-16T07:30:46.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 554, success, outsize: 120 2024-04-16T07:30:46.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.933+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.933+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.933+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.933+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.933+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107042 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 556, success, outsize: 120 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.955+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.955+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.955+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.955+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.955+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107133 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 558, success, outsize: 120 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.975+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.975+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.975+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.975+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.975+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107230 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 560, success, outsize: 120 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.998+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.998+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.998+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.998+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:46.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:46.998+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107324 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 562, success, outsize: 120 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.022+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.022+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.022+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.022+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.022+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107412 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 564, success, outsize: 120 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.045+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.045+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.045+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.045+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.045+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107503 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 566, success, outsize: 120 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.069+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.069+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.069+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.069+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.069+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107600 2024-04-16T07:30:47.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 568, success, outsize: 120 2024-04-16T07:30:47.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.095+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.095+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.095+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.095+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.095+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107694 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 570, success, outsize: 120 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.119+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.119+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.119+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.119+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.119+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:47.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:47.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:47.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:47.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:47.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:47.128+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107782 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 572, success, outsize: 120 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.141+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.141+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.141+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.141+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.141+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107873 2024-04-16T07:30:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 574, success, outsize: 120 2024-04-16T07:30:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.164+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.164+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.164+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.164+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.164+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107970 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 576, success, outsize: 120 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.188+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.188+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.188+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.188+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.188+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108064 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 578, success, outsize: 120 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.210+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.210+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.210+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.210+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.210+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108152 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 580, success, outsize: 120 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.229+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.229+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.229+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.229+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.229+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108243 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 582, success, outsize: 120 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.250+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.250+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.250+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.250+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.250+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108340 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 584, success, outsize: 120 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.274+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.274+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.274+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:47.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.274+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:47.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.274+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:47.281 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:47 smithi104 ceph-mon[81027]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:47.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:47.414+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:47.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:47 smithi026 ceph-mon[92215]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:48.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:48.127+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:48.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:48.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:48.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:48.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:48.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:48.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:48.414+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.127+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.128+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.128+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.128+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.128+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:49.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:49.128+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:30:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi104.zwhiwr"]}]: dispatch 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:49.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:49.414+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:30:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:49.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi104.zwhiwr"]}]: dispatch 2024-04-16T07:30:50.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:50 smithi104 ceph-mon[81027]: Detected new or changed devices on smithi104 2024-04-16T07:30:50.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:50 smithi104 ceph-mon[81027]: Upgrade: It appears safe to stop mds.cephfs.smithi104.zwhiwr 2024-04-16T07:30:50.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:50.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:50.128+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:50.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:50 smithi026 ceph-mon[92215]: Detected new or changed devices on smithi104 2024-04-16T07:30:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:50 smithi026 ceph-mon[92215]: Upgrade: It appears safe to stop mds.cephfs.smithi104.zwhiwr 2024-04-16T07:30:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:30:51.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:51.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:51.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:51.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:51.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:51.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:51.129+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:51 smithi104 ceph-mon[81027]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:30:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:51.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:51.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:51 smithi026 ceph-mon[92215]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:30:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.zwhiwr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T07:30:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:52.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:52.128+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:52 smithi104 ceph-mon[81027]: Upgrade: Updating mds.cephfs.smithi104.zwhiwr 2024-04-16T07:30:52.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:52 smithi104 ceph-mon[81027]: Deploying daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:30:52.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:52.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:30:52.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:30:52.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_session(request_renewcaps seq 33) v5 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4011260 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.416+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.104:6826/2349406764 4 ==== client_session(renewcaps seq 33) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:52.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:52.416+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 33) v1 from mds.0 2024-04-16T07:30:52.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:52 smithi026 ceph-mon[92215]: Upgrade: Updating mds.cephfs.smithi104.zwhiwr 2024-04-16T07:30:52.428 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:52 smithi026 ceph-mon[92215]: Deploying daemon mds.cephfs.smithi104.zwhiwr on smithi104 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 234 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000bfd0 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:53.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.129+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:53.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:53 smithi104 ceph-mon[81027]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 290 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4011260 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:53.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:53.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:53 smithi026 ceph-mon[92215]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:53.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.834+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 16 ==== mdsmap(e 31) v2 ==== 583+0+0 (secure 0 0 0) 0x7fe9d0020450 con 0x55fc3f5009a0 2024-04-16T07:30:53.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.834+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 31 2024-04-16T07:30:53.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.834+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 msgr2=0x7fe9b4013640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:53.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.834+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 0x7fe9b4013640 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:53.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.834+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 16 ==== mdsmap(e 31) v2 ==== 583+0+0 (secure 0 0 0) 0x7fbf8c0198e0 con 0x7fbf70008390 2024-04-16T07:30:53.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.834+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 31 2024-04-16T07:30:53.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.834+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 msgr2=0x7fbf7000c3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-16T07:30:53.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.834+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 0x7fbf7000c3a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 17 ==== mdsmap(e 32) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fe9d00209b0 con 0x55fc3f5009a0 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 32 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 1 client.24339 mds incarnation changed from 0 to 32 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 msgr2=0x7fe9b4013640 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fe9b4011260 0x7fe9b4013640 unknown :-1 s=CLOSED pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:53.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.846+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 17 ==== mdsmap(e 32) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fbf8c0198e0 con 0x7fbf70008390 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 32 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 1 client.14524 mds incarnation changed from 0 to 32 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 msgr2=0x7fbf7000c3a0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:6826/2349406764,v1:172.21.15.104:6827/2349406764] conn(0x7fbf7000bfd0 0x7fbf7000c3a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.845+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:53.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108747 2024-04-16T07:30:53.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 586, success, outsize: 120 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.853+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.853+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.853+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.853+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:53.853+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126344 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 474, success, outsize: 120 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.853+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.853+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.853+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:53.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.853+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:53.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:53.853+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: Standby daemon mds.cephfs.smithi026.kqpyuh assigned to filesystem cephfs as rank 0 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:54 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:replay} 2 up:standby 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:54.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.129+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:53 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[92211]: 2024-04-16T07:30:53.829+0000 7fbd50600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T07:30:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T07:30:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T07:30:54.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: Standby daemon mds.cephfs.smithi026.kqpyuh assigned to filesystem cephfs as rank 0 2024-04-16T07:30:54.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T07:30:54.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T07:30:54.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:54 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:replay} 2 up:standby 2024-04-16T07:30:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:54.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:54.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 18 ==== mdsmap(e 33) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fe9d00209b0 con 0x55fc3f5009a0 2024-04-16T07:30:54.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 33 2024-04-16T07:30:54.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:54.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 10 client.24339 send_reconnect to mds.0 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 20 client.24339 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 10 client.24339 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 10 client.24339 path #0x1 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 10 client.24339 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:54.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.849+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fe9c0051580 con 0x7fe9b4014240 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.851+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 18 ==== mdsmap(e 33) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fbf8c0198e0 con 0x7fbf70008390 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 33 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 10 client.14524 send_reconnect to mds.0 2024-04-16T07:30:54.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 20 client.14524 trim_cache_for_reconnect mds.0 trimmed 0 dentries 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 10 client.14524 caps on 0x1.head pAsLsXsFs wants - 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 10 client.14524 path #0x1 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 10 client.14524 snaprealm snaprealm(0x1 nref=1 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=2024-04-16T07:19:43.615094+0000 change_attr=0) 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.849+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_reconnect(1 caps 1 realms ) v5 -- 0x7fbf84016810 con 0x7fbf7000cfc0 2024-04-16T07:30:54.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.850+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.853+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.26:6826/2688223128 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.853+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dd95d1f1b7dd2f1 server_cookie=fbcd016cb10104ee in_seq=0 out_seq=0 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.852+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38829621a8938918 server_cookie=f2dcf3fac4bbc2cf in_seq=0 out_seq=0 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.852+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.26:6826/2688223128 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.853+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 1 ==== client_session(open) v7 ==== 113+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.853+0000 7fbf77400640 10 client.14524 handle_client_session client_session(open) v7 from mds.0 2024-04-16T07:30:54.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.853+0000 7fbf77400640 10 client.14524 mds.0 already opened, ignore it 2024-04-16T07:30:54.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.853+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 1 ==== client_session(open) v7 ==== 113+0+0 (crc 0 0 0) 0x7fe9c804c040 con 0x7fe9b4014240 2024-04-16T07:30:54.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.853+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(open) v7 from mds.0 2024-04-16T07:30:54.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.853+0000 7fe9baa00640 10 client.24339 mds.0 already opened, ignore it 2024-04-16T07:30:54.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108748 2024-04-16T07:30:54.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 588, success, outsize: 120 2024-04-16T07:30:54.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.857+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:30:54.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.857+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.857+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.857+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:54.857+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126356 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 476, success, outsize: 120 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.857+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.857+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.857+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.857+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:30:54.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:54.857+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:55.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.129+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:55.332 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:55 smithi026 ceph-mon[92215]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:55.332 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:55 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:reconnect 2024-04-16T07:30:55.332 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:55 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:reconnect} 2 up:standby 2024-04-16T07:30:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:55 smithi104 ceph-mon[81027]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:30:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:55 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:reconnect 2024-04-16T07:30:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:55 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:reconnect} 2 up:standby 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 -- 172.21.15.26:0/257858352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2840708f0 msgr2=0x7ff284070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 --2- 172.21.15.26:0/257858352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2840708f0 0x7ff284070d60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff270004940 tx=0x7ff270009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 -- 172.21.15.26:0/257858352 shutdown_connections 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 --2- 172.21.15.26:0/257858352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff2840708f0 0x7ff284070d60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 --2- 172.21.15.26:0/257858352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff2840703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 -- 172.21.15.26:0/257858352 >> 172.21.15.26:0/257858352 conn(0x7ff2840f8580 msgr2=0x7ff2840fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:55.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 -- 172.21.15.26:0/257858352 shutdown_connections 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.332+0000 7ff28aa00700 1 -- 172.21.15.26:0/257858352 wait complete. 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.333+0000 7ff28aa00700 1 Processor -- start 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.333+0000 7ff28aa00700 1 -- start start 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff28aa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff28410fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff28aa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff28aa00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff284110750 con 0x7ff284110290 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff28aa00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841108c0 con 0x7ff284071ea0 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff283400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff283400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53354/0 (socket says 172.21.15.26:53354) 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff283400700 1 -- 172.21.15.26:0/1097412093 learned_addr learned my addr 172.21.15.26:0/1097412093 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.334+0000 7ff283e00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff28410fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff283400700 1 -- 172.21.15.26:0/1097412093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 msgr2=0x7ff28410fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff283400700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff28410fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff283400700 1 -- 172.21.15.26:0/1097412093 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270005040 con 0x7ff284110290 2024-04-16T07:30:55.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff283e00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff28410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:30:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff283400700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff27000b970 tx=0x7ff270009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27000e040 con 0x7ff284110290 2024-04-16T07:30:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff270004290 con 0x7ff284110290 2024-04-16T07:30:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27001c3f0 con 0x7ff284110290 2024-04-16T07:30:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2841157c0 con 0x7ff284110290 2024-04-16T07:30:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.335+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff284115d50 con 0x7ff284110290 2024-04-16T07:30:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.337+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28404c710 con 0x7ff284110290 2024-04-16T07:30:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.338+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff27000bbf0 con 0x7ff284110290 2024-04-16T07:30:55.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.339+0000 7ff27be00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 0x7ff2680710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.339+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7ff2700906d0 con 0x7ff284110290 2024-04-16T07:30:55.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.339+0000 7ff283e00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 0x7ff2680710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.340+0000 7ff283e00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 0x7ff2680710b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff274007fd0 tx=0x7ff274009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:55.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.343+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff2700593d0 con 0x7ff284110290 2024-04-16T07:30:55.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:55.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:55.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:55.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:55.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:55.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:55.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.597+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff284103fd0 con 0x7ff26806ec00 2024-04-16T07:30:55.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.600+0000 7ff27be00700 1 -- 172.21.15.26:0/1097412093 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff284103fd0 con 0x7ff26806ec00 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 msgr2=0x7ff2680710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 0x7ff2680710b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff274007fd0 tx=0x7ff274009040 comp rx=0 tx=0).stop 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 msgr2=0x7ff284115280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff27000b970 tx=0x7ff270009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 shutdown_connections 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff26806ec00 0x7ff2680710b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff284071ea0 0x7ff28410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 --2- 172.21.15.26:0/1097412093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff284110290 0x7ff284115280 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.603+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 >> 172.21.15.26:0/1097412093 conn(0x7ff2840f8580 msgr2=0x7ff2840f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.604+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 shutdown_connections 2024-04-16T07:30:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.604+0000 7ff28aa00700 1 -- 172.21.15.26:0/1097412093 wait complete. 2024-04-16T07:30:55.619 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:30:55.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.766+0000 7fcf78400700 1 -- 172.21.15.26:0/2495132392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fe040 msgr2=0x7fcf700fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.766+0000 7fcf78400700 1 --2- 172.21.15.26:0/2495132392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fe040 0x7fcf700fe4b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c004940 tx=0x7fcf5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:55.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 -- 172.21.15.26:0/2495132392 shutdown_connections 2024-04-16T07:30:55.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 --2- 172.21.15.26:0/2495132392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fe040 0x7fcf700fe4b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 --2- 172.21.15.26:0/2495132392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fce40 0x7fcf700fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 -- 172.21.15.26:0/2495132392 >> 172.21.15.26:0/2495132392 conn(0x7fcf700f8530 msgr2=0x7fcf700fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:55.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 -- 172.21.15.26:0/2495132392 shutdown_connections 2024-04-16T07:30:55.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.767+0000 7fcf78400700 1 -- 172.21.15.26:0/2495132392 wait complete. 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 Processor -- start 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 -- start start 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fe040 0x7fcf7010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7010e580 con 0x7fcf700fce40 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.768+0000 7fcf78400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7010edd0 con 0x7fcf700fe040 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53378/0 (socket says 172.21.15.26:53378) 2024-04-16T07:30:55.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 -- 172.21.15.26:0/854589915 learned_addr learned my addr 172.21.15.26:0/854589915 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:55.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75000700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fe040 0x7fcf7010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 -- 172.21.15.26:0/854589915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fe040 msgr2=0x7fcf7010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:55.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fe040 0x7fcf7010df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:55.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.769+0000 7fcf75a00700 1 -- 172.21.15.26:0/854589915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c005040 con 0x7fcf700fce40 2024-04-16T07:30:55.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf75a00700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcf64002a80 tx=0x7fcf64004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf64007cc0 con 0x7fcf700fce40 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf6400f040 con 0x7fcf700fce40 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7010f0b0 con 0x7fcf700fce40 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf6400dad0 con 0x7fcf700fce40 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.770+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7010f5f0 con 0x7fcf700fce40 2024-04-16T07:30:55.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.771+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7005f5f0 con 0x7fcf700fce40 2024-04-16T07:30:55.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.772+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf6400dc30 con 0x7fcf700fce40 2024-04-16T07:30:55.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.773+0000 7fcf6d600700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 0x7fcf580710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:55.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.773+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fcf6408e7e0 con 0x7fcf700fce40 2024-04-16T07:30:55.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.774+0000 7fcf75000700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 0x7fcf580710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:55.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.774+0000 7fcf75000700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 0x7fcf580710b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b3b0 tx=0x7fcf5c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:55.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:55.779+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcf6405af10 con 0x7fcf700fce40 2024-04-16T07:30:55.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.851+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 19 ==== mdsmap(e 34) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fe9d00209b0 con 0x55fc3f5009a0 2024-04-16T07:30:55.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.851+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 34 2024-04-16T07:30:55.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.852+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 19 ==== mdsmap(e 34) v2 ==== 1254+0+0 (secure 0 0 0) 0x7fbf8c0198e0 con 0x7fbf70008390 2024-04-16T07:30:55.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.852+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 34 2024-04-16T07:30:55.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fbf84016810 con 0x7fbf7000cfc0 2024-04-16T07:30:55.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 10 client.14524 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:55.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 10 client.14524 mds.0 seq now 1 2024-04-16T07:30:55.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 10 client.14524 update_snap_trace len 70 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0x7fbf7000bcb0, nref was 1 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 20 client.14524 get_snap_realm 0x3 0x7fbf7000bcb0, nref now is 2 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 10 client.14524 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:55.860+0000 7fbf77400640 20 client.14524 put_snap_realm 0x3 0x7fbf7000bcb0 2 -> 1 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 2 ==== client_snap(update split=0x3 tracelen=70) v1 ==== 94+0+0 (crc 0 0 0) 0x7fe9c0051580 con 0x7fe9b4014240 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 10 client.24339 handle_snap client_snap(update split=0x3 tracelen=70) v1 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 10 client.24339 mds.0 seq now 1 2024-04-16T07:30:55.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 10 client.24339 update_snap_trace len 70 2024-04-16T07:30:55.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0x7fe9b4010fc0, nref was 1 2024-04-16T07:30:55.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 20 client.24339 get_snap_realm 0x3 0x7fe9b4010fc0, nref now is 2 2024-04-16T07:30:55.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 10 client.24339 update_snap_trace snaprealm(0x3 nref=2 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[] last_modified=0.000000 change_attr=0) seq 1 <= 1 and same parent, SKIPPING 2024-04-16T07:30:55.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:55.859+0000 7fe9baa00640 20 client.24339 put_snap_realm 0x3 0x7fe9b4010fc0 2 -> 1 2024-04-16T07:30:56.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.031+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf7010f9d0 con 0x7fcf5806ec00 2024-04-16T07:30:56.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.035+0000 7fcf6d600700 1 -- 172.21.15.26:0/854589915 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcf7010f9d0 con 0x7fcf5806ec00 2024-04-16T07:30:56.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.038+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 msgr2=0x7fcf580710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.038+0000 7fcf78400700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 0x7fcf580710b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b3b0 tx=0x7fcf5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:56.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.038+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 msgr2=0x7fcf7010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.038+0000 7fcf78400700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcf64002a80 tx=0x7fcf64004600 comp rx=0 tx=0).stop 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 shutdown_connections 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcf5806ec00 0x7fcf580710b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf700fce40 0x7fcf7010da40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 --2- 172.21.15.26:0/854589915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf700fe040 0x7fcf7010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 >> 172.21.15.26:0/854589915 conn(0x7fcf700f8530 msgr2=0x7fcf700f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 shutdown_connections 2024-04-16T07:30:56.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.039+0000 7fcf78400700 1 -- 172.21.15.26:0/854589915 wait complete. 2024-04-16T07:30:56.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:56.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:56.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:56.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 5 client.14524 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:56.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:56.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.129+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 -- 172.21.15.26:0/4176096637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c0ff000 msgr2=0x7fc49c0ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 --2- 172.21.15.26:0/4176096637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c0ff000 0x7fc49c0ff410 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc48c004940 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 -- 172.21.15.26:0/4176096637 shutdown_connections 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 --2- 172.21.15.26:0/4176096637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c1002b0 0x7fc49c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 --2- 172.21.15.26:0/4176096637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c0ff000 0x7fc49c0ff410 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.206+0000 7fc4a1600700 1 -- 172.21.15.26:0/4176096637 >> 172.21.15.26:0/4176096637 conn(0x7fc49c0fa790 msgr2=0x7fc49c0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.207+0000 7fc4a1600700 1 -- 172.21.15.26:0/4176096637 shutdown_connections 2024-04-16T07:30:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.207+0000 7fc4a1600700 1 -- 172.21.15.26:0/4176096637 wait complete. 2024-04-16T07:30:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 Processor -- start 2024-04-16T07:30:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 -- start start 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0ff000 0x7fc49c10fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c110620 con 0x7fc49c1002b0 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.208+0000 7fc4a1600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c110790 con 0x7fc49c0ff000 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53406/0 (socket says 172.21.15.26:53406) 2024-04-16T07:30:56.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 -- 172.21.15.26:0/3545847166 learned_addr learned my addr 172.21.15.26:0/3545847166 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:56.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49aa00700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0ff000 0x7fc49c10fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:56.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 -- 172.21.15.26:0/3545847166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0ff000 msgr2=0x7fc49c10fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0ff000 0x7fc49c10fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.209+0000 7fc49a000700 1 -- 172.21.15.26:0/3545847166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c005040 con 0x7fc49c1002b0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc49a000700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc48800eb40 tx=0x7fc48800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc488007cc0 con 0x7fc49c1002b0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc48800f040 con 0x7fc49c1002b0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc488009620 con 0x7fc49c1002b0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c115190 con 0x7fc49c1002b0 2024-04-16T07:30:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.210+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c1156d0 con 0x7fc49c1002b0 2024-04-16T07:30:56.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.211+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c04c710 con 0x7fc49c1002b0 2024-04-16T07:30:56.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.212+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc488009780 con 0x7fc49c1002b0 2024-04-16T07:30:56.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.213+0000 7fc492a00700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 0x7fc4800710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.214+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fc48808e650 con 0x7fc49c1002b0 2024-04-16T07:30:56.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.214+0000 7fc49aa00700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 0x7fc4800710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:56.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.215+0000 7fc49aa00700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 0x7fc4800710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00b0d0 tx=0x7fc48c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:56.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.219+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc48805aeb0 con 0x7fc49c1002b0 2024-04-16T07:30:56.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:56.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:56.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:56.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 5 client.24339 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric 2024-04-16T07:30:56.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:56.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.415+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:56.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.460+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc49c104550 con 0x7fc48006ec00 2024-04-16T07:30:56.460 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00299995 2024-04-16T07:30:56.460 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00399993 2024-04-16T07:30:56.460 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 613 B/s rd, 0 op/s 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:rejoin 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:rejoin} 2 up:standby 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: daemon mds.cephfs.smithi026.kqpyuh is now active in filesystem cephfs as rank 0 2024-04-16T07:30:56.461 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:56 smithi026 ceph-mon[92215]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.476+0000 7fc492a00700 1 -- 172.21.15.26:0/3545847166 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc49c104550 con 0x7fc48006ec00 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (7m) 43s ago 18m 14.0M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (4m) 43s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (4m) 15s ago 17m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (7m) 43s ago 17m 73.0M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:30:56.476 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (77s) 43s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (49s) 43s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (20s) 15s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (11m) 15s ago 11m 17.2M - 17.2.7-847-g45317f95 b333581ccdf5 0a2cb5728a5e 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (7m) 43s ago 19m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (6m) 15s ago 16m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (6m) 43s ago 20m 49.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (5m) 15s ago 16m 41.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (8m) 43s ago 18m 8061k - 1.5.0 0da6a335fe13 f0d8d4a0d6ee 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 15s ago 17m 7625k - 1.5.0 0da6a335fe13 1c8b469d2559 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (4m) 43s ago 15m 51.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (3m) 43s ago 15m 51.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (3m) 43s ago 14m 43.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (2m) 15s ago 14m 52.4M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (2m) 15s ago 13m 42.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (102s) 15s ago 12m 42.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:30:56.477 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (6m) 43s ago 17m 60.1M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:30:56.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 msgr2=0x7fc4800710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 0x7fc4800710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00b0d0 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 msgr2=0x7fc49c110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc48800eb40 tx=0x7fc48800b040 comp rx=0 tx=0).stop 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 shutdown_connections 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc48006ec00 0x7fc4800710b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0ff000 0x7fc49c10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 --2- 172.21.15.26:0/3545847166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc49c1002b0 0x7fc49c110020 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 >> 172.21.15.26:0/3545847166 conn(0x7fc49c0fa790 msgr2=0x7fc49c0fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 shutdown_connections 2024-04-16T07:30:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.480+0000 7fc4a1600700 1 -- 172.21.15.26:0/3545847166 wait complete. 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: reconnect by client.14524 192.168.0.1:0/2108819864 after 0.00299995 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: reconnect by client.24339 192.168.0.1:0/874387268 after 0.00399993 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 613 B/s rd, 0 op/s 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:rejoin 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: fsmap cephfs:1/1 {0=cephfs.smithi026.kqpyuh=up:rejoin} 2 up:standby 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: daemon mds.cephfs.smithi026.kqpyuh is now active in filesystem cephfs as rank 0 2024-04-16T07:30:56.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:56 smithi104 ceph-mon[81027]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:56.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.641+0000 7f0624600700 1 -- 172.21.15.26:0/1737511202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 msgr2=0x7f061c072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.641+0000 7f0624600700 1 --2- 172.21.15.26:0/1737511202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c072c30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0610004940 tx=0x7f0610009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:56.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 -- 172.21.15.26:0/1737511202 shutdown_connections 2024-04-16T07:30:56.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 --2- 172.21.15.26:0/1737511202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f061c073170 0x7f061c073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 --2- 172.21.15.26:0/1737511202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c072c30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 -- 172.21.15.26:0/1737511202 >> 172.21.15.26:0/1737511202 conn(0x7f061c0fa950 msgr2=0x7f061c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 -- 172.21.15.26:0/1737511202 shutdown_connections 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.642+0000 7f0624600700 1 -- 172.21.15.26:0/1737511202 wait complete. 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.643+0000 7f0624600700 1 Processor -- start 2024-04-16T07:30:56.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.643+0000 7f0624600700 1 -- start start 2024-04-16T07:30:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0624600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0624600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f061c1145f0 0x7f061c115440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0624600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c114ab0 con 0x7f061c072820 2024-04-16T07:30:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0624600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c114bf0 con 0x7f061c1145f0 2024-04-16T07:30:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0621c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:56.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0621c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53424/0 (socket says 172.21.15.26:53424) 2024-04-16T07:30:56.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.644+0000 7f0621c00700 1 -- 172.21.15.26:0/4006418697 learned_addr learned my addr 172.21.15.26:0/4006418697 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0621c00700 1 -- 172.21.15.26:0/4006418697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f061c1145f0 msgr2=0x7f061c115440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0621c00700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f061c1145f0 0x7f061c115440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0621c00700 1 -- 172.21.15.26:0/4006418697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0610005040 con 0x7f061c072820 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0621c00700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f061000b970 tx=0x7f0610009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f061000e040 con 0x7f061c072820 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0610004290 con 0x7f061c072820 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c115a40 con 0x7f061c072820 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.645+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f061001c630 con 0x7f061c072820 2024-04-16T07:30:56.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.646+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c115fd0 con 0x7f061c072820 2024-04-16T07:30:56.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.647+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c05f5f0 con 0x7f061c072820 2024-04-16T07:30:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.648+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f061000bbf0 con 0x7f061c072820 2024-04-16T07:30:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.650+0000 7f0619600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 0x7f06040710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.650+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f061008fcd0 con 0x7f061c072820 2024-04-16T07:30:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.650+0000 7f0621200700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 0x7f06040710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.651+0000 7f0621200700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 0x7f06040710c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0608004d00 tx=0x7f0608009ce0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:56.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.656+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f061005c500 con 0x7f061c072820 2024-04-16T07:30:56.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.854+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 20 ==== mdsmap(e 35) v2 ==== 1262+0+0 (secure 0 0 0) 0x7fbf8c0198e0 con 0x7fbf70008390 2024-04-16T07:30:56.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.855+0000 7fbf77400640 1 client.14524 handle_mds_map epoch 35 2024-04-16T07:30:56.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.855+0000 7fbf77400640 10 client.14524 kick_requests for mds.0 2024-04-16T07:30:56.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.855+0000 7fbf77400640 10 client.14524 kick_flushing_caps mds.0 2024-04-16T07:30:56.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:56.855+0000 7fbf77400640 10 client.14524 connect_mds_targets for mds.0 2024-04-16T07:30:56.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.854+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 20 ==== mdsmap(e 35) v2 ==== 1262+0+0 (secure 0 0 0) 0x7fe9d00209b0 con 0x55fc3f5009a0 2024-04-16T07:30:56.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.854+0000 7fe9baa00640 1 client.24339 handle_mds_map epoch 35 2024-04-16T07:30:56.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.854+0000 7fe9baa00640 10 client.24339 kick_requests for mds.0 2024-04-16T07:30:56.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.854+0000 7fe9baa00640 10 client.24339 kick_flushing_caps mds.0 2024-04-16T07:30:56.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:56.854+0000 7fe9baa00640 10 client.24339 connect_mds_targets for mds.0 2024-04-16T07:30:56.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.984+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f061c10bf10 con 0x7f061c072820 2024-04-16T07:30:56.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.985+0000 7f0619600700 1 -- 172.21.15.26:0/4006418697 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0610013e60 con 0x7f061c072820 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:30:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 13 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:30:56.986 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:30:56.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 msgr2=0x7f06040710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 0x7f06040710c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0608004d00 tx=0x7f0608009ce0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 msgr2=0x7f061c1140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f061000b970 tx=0x7f0610009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 shutdown_connections 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f060406ec10 0x7f06040710c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f061c072820 0x7f061c1140b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 --2- 172.21.15.26:0/4006418697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f061c1145f0 0x7f061c115440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 >> 172.21.15.26:0/4006418697 conn(0x7f061c0fa950 msgr2=0x7f061c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 shutdown_connections 2024-04-16T07:30:56.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:56.989+0000 7f0624600700 1 -- 172.21.15.26:0/4006418697 wait complete. 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.129+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.129+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.129+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.129+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.130+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:57.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:57.130+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 -- 172.21.15.26:0/3558441009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180073170 msgr2=0x7f9180073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 --2- 172.21.15.26:0/3558441009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180073170 0x7f9180073b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9168004940 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 -- 172.21.15.26:0/3558441009 shutdown_connections 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 --2- 172.21.15.26:0/3558441009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180073170 0x7f9180073b80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 --2- 172.21.15.26:0/3558441009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9180072820 0x7f9180072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 -- 172.21.15.26:0/3558441009 >> 172.21.15.26:0/3558441009 conn(0x7f91800fa950 msgr2=0x7f91800fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 -- 172.21.15.26:0/3558441009 shutdown_connections 2024-04-16T07:30:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.152+0000 7f9185c00700 1 -- 172.21.15.26:0/3558441009 wait complete. 2024-04-16T07:30:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.153+0000 7f9185c00700 1 Processor -- start 2024-04-16T07:30:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.153+0000 7f9185c00700 1 -- start start 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.153+0000 7f9185c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f9185c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91801144b0 0x7f91801194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f9185c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180114a00 con 0x7f9180072820 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f9185c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180114b40 con 0x7f91801144b0 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53454/0 (socket says 172.21.15.26:53454) 2024-04-16T07:30:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 -- 172.21.15.26:0/2866517425 learned_addr learned my addr 172.21.15.26:0/2866517425 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 -- 172.21.15.26:0/2866517425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91801144b0 msgr2=0x7f91801194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91801144b0 0x7f91801194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.154+0000 7f917ea00700 1 -- 172.21.15.26:0/2866517425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168005040 con 0x7f9180072820 2024-04-16T07:30:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f917ea00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f917000b790 tx=0x7f9170009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917000f6f0 con 0x7f9180072820 2024-04-16T07:30:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f917000fd20 con 0x7f9180072820 2024-04-16T07:30:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91700103a0 con 0x7f9180072820 2024-04-16T07:30:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180119ae0 con 0x7f9180072820 2024-04-16T07:30:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.155+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918011a0a0 con 0x7f9180072820 2024-04-16T07:30:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.156+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918004c710 con 0x7f9180072820 2024-04-16T07:30:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.157+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9170011400 con 0x7f9180072820 2024-04-16T07:30:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.158+0000 7f9176a00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 0x7f9164071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.158+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f917008e7e0 con 0x7f9180072820 2024-04-16T07:30:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.159+0000 7f9177e00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 0x7f9164071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.159+0000 7f9177e00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 0x7f9164071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9168004f90 tx=0x7f9168009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:57.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.163+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f917005af90 con 0x7f9180072820 2024-04-16T07:30:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: Cluster is now healthy 2024-04-16T07:30:57.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:active 2024-04-16T07:30:57.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:30:57.262 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:57 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4006418697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:57.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:57.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:57.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:57.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:30:57.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:57.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:57.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.445+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f918011a320 con 0x7f9180072820 2024-04-16T07:30:57.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.446+0000 7f9176a00700 1 -- 172.21.15.26:0/2866517425 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1761 (secure 0 0 0) 0x7f917005ab20 con 0x7f9180072820 2024-04-16T07:30:57.445 INFO:teuthology.orchestra.run.smithi026.stdout:e35 2024-04-16T07:30:57.445 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:30:57.445 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:30:57.445 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 35 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:56.852323+0000 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:30:57.446 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 85 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=34404} 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 34404 members: 34404 2024-04-16T07:30:57.447 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:34404} state up:active seq 22 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:30:57.448 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 msgr2=0x7f9164071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 0x7f9164071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9168004f90 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 msgr2=0x7f9180113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f917000b790 tx=0x7f9170009220 comp rx=0 tx=0).stop 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 shutdown_connections 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f916406ecd0 0x7f9164071180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9180072820 0x7f9180113f70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 --2- 172.21.15.26:0/2866517425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91801144b0 0x7f91801194e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.449+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 >> 172.21.15.26:0/2866517425 conn(0x7f91800fa950 msgr2=0x7f91800fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.450+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 shutdown_connections 2024-04-16T07:30:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.450+0000 7f9185c00700 1 -- 172.21.15.26:0/2866517425 wait complete. 2024-04-16T07:30:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 35 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: Cluster is now healthy 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] up:active 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 2 up:standby 2024-04-16T07:30:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:57 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4006418697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 -- 172.21.15.26:0/992919135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb2881046d0 msgr2=0x7fb288104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 --2- 172.21.15.26:0/992919135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb2881046d0 0x7fb288104b40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb270004940 tx=0x7fb270009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 -- 172.21.15.26:0/992919135 shutdown_connections 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 --2- 172.21.15.26:0/992919135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb2881046d0 0x7fb288104b40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 --2- 172.21.15.26:0/992919135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb288103420 0x7fb288103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 -- 172.21.15.26:0/992919135 >> 172.21.15.26:0/992919135 conn(0x7fb288072640 msgr2=0x7fb288074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 -- 172.21.15.26:0/992919135 shutdown_connections 2024-04-16T07:30:57.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.604+0000 7fb28de00700 1 -- 172.21.15.26:0/992919135 wait complete. 2024-04-16T07:30:57.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.605+0000 7fb28de00700 1 Processor -- start 2024-04-16T07:30:57.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.605+0000 7fb28de00700 1 -- start start 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb28de00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb28de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2881046d0 0x7fb288114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb28de00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb288114a90 con 0x7fb288103420 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb28de00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb288114c00 con 0x7fb2881046d0 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53472/0 (socket says 172.21.15.26:53472) 2024-04-16T07:30:57.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 -- 172.21.15.26:0/723302955 learned_addr learned my addr 172.21.15.26:0/723302955 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:57.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 -- 172.21.15.26:0/723302955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2881046d0 msgr2=0x7fb288114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2881046d0 0x7fb288114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.606+0000 7fb286a00700 1 -- 172.21.15.26:0/723302955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270005040 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb286a00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c830 tx=0x7fb278004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb27800a760 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb27800ad90 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb288119660 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2780113a0 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.607+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288119c20 con 0x7fb288103420 2024-04-16T07:30:57.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.608+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28804c710 con 0x7fb288103420 2024-04-16T07:30:57.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.609+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb278012400 con 0x7fb288103420 2024-04-16T07:30:57.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.610+0000 7fb27ea00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 0x7fb26c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:57.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.610+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fb27808e9c0 con 0x7fb288103420 2024-04-16T07:30:57.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.611+0000 7fb27fe00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 0x7fb26c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:57.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.611+0000 7fb27fe00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 0x7fb26c0710b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb270004f90 tx=0x7fb270009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.615+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb27805b1f0 con 0x7fb288103420 2024-04-16T07:30:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.860+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb288108a10 con 0x7fb26c06ec00 2024-04-16T07:30:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.862+0000 7fb27ea00700 1 -- 172.21.15.26:0/723302955 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb288108a10 con 0x7fb26c06ec00 2024-04-16T07:30:57.861 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:30:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:30:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "15/21 daemons upgraded", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:30:57.862 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 msgr2=0x7fb26c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 0x7fb26c0710b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb270004f90 tx=0x7fb270009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 msgr2=0x7fb288113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c830 tx=0x7fb278004470 comp rx=0 tx=0).stop 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 shutdown_connections 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb26c06ec00 0x7fb26c0710b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb288103420 0x7fb288113f50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 --2- 172.21.15.26:0/723302955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2881046d0 0x7fb288114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.864+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 >> 172.21.15.26:0/723302955 conn(0x7fb288072640 msgr2=0x7fb288073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.865+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 shutdown_connections 2024-04-16T07:30:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:57.865+0000 7fb28de00700 1 -- 172.21.15.26:0/723302955 wait complete. 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 -- 172.21.15.26:0/3457523484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b0066c40 msgr2=0x7ff7b0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 --2- 172.21.15.26:0/3457523484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b0066c40 0x7ff7b0101ca0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0004940 tx=0x7ff7a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 -- 172.21.15.26:0/3457523484 shutdown_connections 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 --2- 172.21.15.26:0/3457523484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b01021e0 0x7ff7b01045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 --2- 172.21.15.26:0/3457523484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b0066c40 0x7ff7b0101ca0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 -- 172.21.15.26:0/3457523484 >> 172.21.15.26:0/3457523484 conn(0x7ff7b00f7290 msgr2=0x7ff7b00f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 -- 172.21.15.26:0/3457523484 shutdown_connections 2024-04-16T07:30:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.021+0000 7ff7af200700 1 -- 172.21.15.26:0/3457523484 wait complete. 2024-04-16T07:30:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.022+0000 7ff7af200700 1 Processor -- start 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.022+0000 7ff7af200700 1 -- start start 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7af200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b0066c40 0x7ff7b010fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7af200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7af200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0110710 con 0x7ff7b01021e0 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7af200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0110eb0 con 0x7ff7b0066c40 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48296/0 (socket says 172.21.15.26:48296) 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 -- 172.21.15.26:0/3941655904 learned_addr learned my addr 172.21.15.26:0/3941655904 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ade00700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b0066c40 0x7ff7b010fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 -- 172.21.15.26:0/3941655904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b0066c40 msgr2=0x7ff7b010fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b0066c40 0x7ff7b010fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.023+0000 7ff7ad400700 1 -- 172.21.15.26:0/3941655904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a0005040 con 0x7ff7b01021e0 2024-04-16T07:30:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7ad400700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00b790 tx=0x7ff79c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79c00f770 con 0x7ff7b01021e0 2024-04-16T07:30:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff79c00fda0 con 0x7ff7b01021e0 2024-04-16T07:30:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b0111190 con 0x7ff7b01021e0 2024-04-16T07:30:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79c00d310 con 0x7ff7b01021e0 2024-04-16T07:30:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.024+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b0111750 con 0x7ff7b01021e0 2024-04-16T07:30:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.026+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b00faf30 con 0x7ff7b01021e0 2024-04-16T07:30:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.027+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff79c010420 con 0x7ff7b01021e0 2024-04-16T07:30:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.028+0000 7ff7a6000700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 0x7ff7940710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:30:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.028+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7ff79c08ea80 con 0x7ff7b01021e0 2024-04-16T07:30:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.028+0000 7ff7ade00700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 0x7ff7940710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:30:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.029+0000 7ff7ade00700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 0x7ff7940710b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000b910 tx=0x7ff7a0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:30:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.032+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff79c05b230 con 0x7ff7b01021e0 2024-04-16T07:30:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:30:58.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:58.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:58.130+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:58.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:58 smithi026 ceph-mon[92215]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-16T07:30:58.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:58 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2866517425' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:30:58.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:58 smithi026 ceph-mon[92215]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:58.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.345+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff7b004bed0 con 0x7ff7b01021e0 2024-04-16T07:30:58.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.346+0000 7ff7a6000700 1 -- 172.21.15.26:0/3941655904 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff79c05adc0 con 0x7ff7b01021e0 2024-04-16T07:30:58.345 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 msgr2=0x7ff7940710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 0x7ff7940710b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000b910 tx=0x7ff7a0009e30 comp rx=0 tx=0).stop 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 msgr2=0x7ff7b0110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00b790 tx=0x7ff79c009250 comp rx=0 tx=0).stop 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 shutdown_connections 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff79406ec00 0x7ff7940710b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7b0066c40 0x7ff7b010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 --2- 172.21.15.26:0/3941655904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7b01021e0 0x7ff7b0110080 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:30:58.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.348+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 >> 172.21.15.26:0/3941655904 conn(0x7ff7b00f7290 msgr2=0x7ff7b00f76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:30:58.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.349+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 shutdown_connections 2024-04-16T07:30:58.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:30:58.349+0000 7ff7af200700 1 -- 172.21.15.26:0/3941655904 wait complete. 2024-04-16T07:30:58.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.415+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:58.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:58.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.415+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:58.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.415+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:30:58.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:58.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:58.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:58.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:58 smithi104 ceph-mon[81027]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-16T07:30:58.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:58 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2866517425' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:30:58.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:58 smithi104 ceph-mon[81027]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:30:59.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:30:59.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:30:59.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:30:59.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:30:59.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:30:59.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:30:59.131+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:30:59 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3941655904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:30:59.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:30:59.417+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:30:59.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:30:59 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3941655904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:00.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:00.130+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:00.374 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:00 smithi104 ceph-mon[81027]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-04-16T07:31:00.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:00.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:00.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:00.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:00.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:00.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:00.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:00.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:00 smithi026 ceph-mon[92215]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-04-16T07:31:01.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:01.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:01.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:01.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:01.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:01.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:01.131+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:01.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:01.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:01.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:01.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:01.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:01.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:01.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:02.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:02.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:02.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:02.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:02.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:02.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:02.132+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:02.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:02.416+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:02.548 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:02 smithi104 ceph-mon[81027]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T07:31:02.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:02 smithi026 ceph-mon[92215]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:03.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:03.131+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:03.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:03.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:03.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:03.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:03.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:03.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:03.417+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: mds.? [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] up:boot 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 3 up:standby 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:03.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:03 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:03.774 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: mds.? [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] up:boot 2024-04-16T07:31:03.774 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: fsmap cephfs:1 {0=cephfs.smithi026.kqpyuh=up:active} 3 up:standby 2024-04-16T07:31:03.774 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:31:03.775 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:03.775 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:03.775 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:03 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:04.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:04.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 34) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.131+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.132+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.132+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.132+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 3 ==== client_session(renewcaps seq 34) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf840244e0 con 0x7fbf7000cfc0 2024-04-16T07:31:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:04.132+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 34) v1 from mds.0 2024-04-16T07:31:04.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:04.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:04.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:04.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:04.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:04.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:04.418+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:04 smithi026 ceph-mon[92215]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T07:31:04.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:04 smithi104 ceph-mon[81027]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T07:31:04.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:05.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:05.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:05.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:05.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:05.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:05.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:05.132+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:05.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:05.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:05.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:05.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:05.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:05.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:05.417+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:06.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:06.133+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:06.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:06.417+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:07.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:07.132+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:06 smithi026 ceph-mon[92215]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 177 B/s wr, 7 op/s 2024-04-16T07:31:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:07.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:06 smithi104 ceph-mon[81027]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 177 B/s wr, 7 op/s 2024-04-16T07:31:07.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:07.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:07.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:07.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:07.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:07.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:07.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:07.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:07.419+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:08.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.132+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:08.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.132+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:08.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.132+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:08.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.133+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:08.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.133+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:08.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:08.133+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:08.419+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:09.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:09.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:09.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:09.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:09.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:09.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:09.134+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:08 smithi026 ceph-mon[92215]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-16T07:31:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:09.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:08 smithi104 ceph-mon[81027]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-16T07:31:09.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:09.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:09.419+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:10.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:10.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:10.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:10.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:10.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:10.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:10.134+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:10.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:10.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:10.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:10.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:10.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:10.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:10.419+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:11.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:11.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:11.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:11.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:11.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:11.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:11.133+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:11.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:10 smithi026 ceph-mon[92215]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T07:31:11.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:10 smithi104 ceph-mon[81027]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.419+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.419+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.419+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.420+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:11.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:11.420+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:12.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109721 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 590, success, outsize: 120 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.037+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.037+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.037+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.037+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.037+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109799 2024-04-16T07:31:12.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 592, success, outsize: 120 2024-04-16T07:31:12.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.056+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.056+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.056+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.056+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.056+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109880 2024-04-16T07:31:12.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 594, success, outsize: 120 2024-04-16T07:31:12.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.076+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.076+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.076+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.076+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.076+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109967 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 596, success, outsize: 120 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.095+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.095+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.095+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.095+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.095+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110051 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 598, success, outsize: 120 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.114+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.114+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.114+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.114+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.114+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110129 2024-04-16T07:31:12.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 600, success, outsize: 120 2024-04-16T07:31:12.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.130+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.130+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.130+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.130+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.130+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:12.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:12.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110210 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 602, success, outsize: 120 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.149+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.149+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.149+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.149+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.149+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110297 2024-04-16T07:31:12.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 604, success, outsize: 120 2024-04-16T07:31:12.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.168+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.168+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.168+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.169+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.169+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110381 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 606, success, outsize: 120 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.188+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.188+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.188+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.188+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.188+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110459 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 608, success, outsize: 120 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.207+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.207+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.207+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.207+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.207+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110540 2024-04-16T07:31:12.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 610, success, outsize: 120 2024-04-16T07:31:12.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.225+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.225+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.225+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.225+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.225+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110627 2024-04-16T07:31:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 612, success, outsize: 120 2024-04-16T07:31:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.244+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.244+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.244+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.244+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.244+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110711 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 614, success, outsize: 120 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.264+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.264+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.264+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.264+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.264+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110789 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 616, success, outsize: 120 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.282+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.282+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.282+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.282+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.282+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110870 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 618, success, outsize: 120 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.301+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.301+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.301+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.301+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.301+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110957 2024-04-16T07:31:12.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 620, success, outsize: 120 2024-04-16T07:31:12.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.321+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.321+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.321+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.321+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.321+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111041 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 622, success, outsize: 120 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.342+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.342+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.342+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.342+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.342+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111119 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 624, success, outsize: 120 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.361+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.361+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.361+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.361+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.361+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111200 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 626, success, outsize: 120 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.381+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.381+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.381+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.381+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.381+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111287 2024-04-16T07:31:12.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.403+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 628, success, outsize: 120 2024-04-16T07:31:12.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.403+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.403+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.403+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.403+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 34) v5 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 312 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:12.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:12.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 3 ==== client_session(renewcaps seq 34) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:12.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.420+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 34) v1 from mds.0 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111371 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 630, success, outsize: 120 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.423+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.423+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.423+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.423+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.423+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111449 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 632, success, outsize: 120 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.442+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.442+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.442+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.442+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.442+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111530 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 634, success, outsize: 120 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.462+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.462+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.462+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.462+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.462+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111617 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 636, success, outsize: 120 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.482+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.482+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.482+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.482+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.482+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111701 2024-04-16T07:31:12.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 638, success, outsize: 120 2024-04-16T07:31:12.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.503+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.503+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.503+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.503+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.503+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111779 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 640, success, outsize: 120 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.523+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.523+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.523+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.523+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.523+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111860 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 642, success, outsize: 120 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.543+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.543+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.543+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.543+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.543+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:12.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111947 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 644, success, outsize: 120 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.565+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.565+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.565+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.565+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:12.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:12.565+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:13.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:13.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:13.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:13.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:13.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:13.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:13.134+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:13.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:13 smithi026 ceph-mon[92215]: pgmap v229: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T07:31:13.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:13 smithi104 ceph-mon[81027]: pgmap v229: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:13.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:13.420+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:14.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:14.134+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:14.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:14.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:14.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:14.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:14.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:14.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:14.420+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:31:14.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all mds 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kqpyuh"}]': finished 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kwjewe"}]': finished 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:31:14.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.dpfdje"}]': finished 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.zwhiwr"}]': finished 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all rgw 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all iscsi 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all nfs 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: Upgrade: Setting container_image for all nvmeof 2024-04-16T07:31:14.764 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all mds 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kqpyuh"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kqpyuh"}]': finished 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kwjewe"}]: dispatch 2024-04-16T07:31:14.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi026.kwjewe"}]': finished 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.dpfdje"}]: dispatch 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.dpfdje"}]': finished 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.zwhiwr"}]: dispatch 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi104.zwhiwr"}]': finished 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all rgw 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all iscsi 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all nfs 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: Upgrade: Setting container_image for all nvmeof 2024-04-16T07:31:14.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:15.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:15.134+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.420+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.420+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.420+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.421+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:15.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:15.421+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:16.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.134+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:16.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.134+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:16.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.134+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:16.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.134+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:16.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:16.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:16.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:16.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:16.421+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:16.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:16 smithi026 ceph-mon[92215]: pgmap v231: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:16.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:16 smithi104 ceph-mon[81027]: pgmap v231: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:17.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:17.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:17.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:17.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:17.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:17.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:17.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:17.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:17.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:17.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:17.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:17.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:17.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:17.421+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:17.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:17 smithi026 ceph-mon[92215]: Upgrade: Updating node-exporter.smithi026 (1/2) 2024-04-16T07:31:17.694 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:17 smithi026 ceph-mon[92215]: Deploying daemon node-exporter.smithi026 on smithi026 2024-04-16T07:31:17.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:17 smithi104 ceph-mon[81027]: Upgrade: Updating node-exporter.smithi026 (1/2) 2024-04-16T07:31:17.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:17 smithi104 ceph-mon[81027]: Deploying daemon node-exporter.smithi026 on smithi026 2024-04-16T07:31:18.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:18.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:18.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:18.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:18.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:18.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:18.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:18.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:18.421+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:18.665 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:18 smithi026 ceph-mon[92215]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:18 smithi104 ceph-mon[81027]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:19.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:19.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:19.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:19.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:19.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:19.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:19.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:19.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:19.422+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:19.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:19.706 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:20.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:20.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:20.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:20.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:20.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:20.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:20.135+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:20.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:20.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:20.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:20.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:20.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:20.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:20.422+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:20.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:20 smithi026 ceph-mon[92215]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:20 smithi104 ceph-mon[81027]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:21.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:21.136+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:21.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:21.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:21.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:21.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:21.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:21.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:21.422+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:21.935 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:21.935 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:21 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:21 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:22.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:22.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:22.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:22.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:22.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:22.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:22.136+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.422+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.422+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.422+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.423+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:22.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:22.423+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:22.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:22 smithi104 ceph-mon[81027]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:23.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:22 smithi026 ceph-mon[92215]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:23.136+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:23.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:23.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:23.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:23.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:23.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:23.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:23.423+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.136+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:31:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:31:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:31:24.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 35) v5 -- 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.137+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.138+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 4 ==== client_session(renewcaps seq 35) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:31:24.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:24.138+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 35) v1 from mds.0 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:24.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:24.423+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:24 smithi026 ceph-mon[92215]: Upgrade: Updating node-exporter.smithi104 (2/2) 2024-04-16T07:31:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:24 smithi026 ceph-mon[92215]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-16T07:31:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:24 smithi026 ceph-mon[92215]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:25.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:24 smithi104 ceph-mon[81027]: Upgrade: Updating node-exporter.smithi104 (2/2) 2024-04-16T07:31:25.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:24 smithi104 ceph-mon[81027]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-16T07:31:25.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:24 smithi104 ceph-mon[81027]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:25.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:25.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:25.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:25.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:25.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:25.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:25.137+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:25.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:25.423+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:26.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:26.137+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.423+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.423+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.423+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.424+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:26.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:26.424+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:26.949 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:26 smithi104 ceph-mon[81027]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:27.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:26 smithi026 ceph-mon[92215]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:27.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:27.137+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:27.424+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:28.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:28.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:28.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:28.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:28.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:28.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:28.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:28.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:28.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:28.424+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:28.448 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:28.448 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:28.448 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:28 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:28.448 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:28 smithi104 ceph-mon[81027]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:28.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:28.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:28.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:28 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:28.506 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:28 smithi026 ceph-mon[92215]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 -- 172.21.15.26:0/4273277134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 msgr2=0x7f803c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 --2- 172.21.15.26:0/4273277134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c102390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8024004940 tx=0x7f8024009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 -- 172.21.15.26:0/4273277134 shutdown_connections 2024-04-16T07:31:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 --2- 172.21.15.26:0/4273277134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c102390 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 --2- 172.21.15.26:0/4273277134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 0x7f803c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 -- 172.21.15.26:0/4273277134 >> 172.21.15.26:0/4273277134 conn(0x7f803c0f74c0 msgr2=0x7f803c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 -- 172.21.15.26:0/4273277134 shutdown_connections 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.516+0000 7f8041e00700 1 -- 172.21.15.26:0/4273277134 wait complete. 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.517+0000 7f8041e00700 1 Processor -- start 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.517+0000 7f8041e00700 1 -- start start 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.517+0000 7f8041e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 0x7f803c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8041e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8041e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c10e510 con 0x7f803c0fff70 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8041e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c10ed60 con 0x7f803c0662f0 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8033e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8033e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38924/0 (socket says 172.21.15.26:38924) 2024-04-16T07:31:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f8033e00700 1 -- 172.21.15.26:0/1673899836 learned_addr learned my addr 172.21.15.26:0/1673899836 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.518+0000 7f803aa00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 0x7f803c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8033e00700 1 -- 172.21.15.26:0/1673899836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 msgr2=0x7f803c10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8033e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 0x7f803c10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8033e00700 1 -- 172.21.15.26:0/1673899836 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024005040 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8033e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8024013870 tx=0x7f8024009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f802400e070 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8024013de0 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c10efe0 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.519+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f802401cea0 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.520+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c10f4c0 con 0x7f803c0fff70 2024-04-16T07:31:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.521+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c05f5f0 con 0x7f803c0fff70 2024-04-16T07:31:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.522+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f802401c690 con 0x7f803c0fff70 2024-04-16T07:31:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.523+0000 7f8032a00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 0x7f80200710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.524+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f8024090060 con 0x7f803c0fff70 2024-04-16T07:31:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.524+0000 7f803aa00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 0x7f80200710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.525+0000 7f803aa00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 0x7f80200710c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f802c009b80 tx=0x7f802c00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.529+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f802405c890 con 0x7f803c0fff70 2024-04-16T07:31:28.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.797+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f803c10f7f0 con 0x7f802006ec10 2024-04-16T07:31:28.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.799+0000 7f8032a00700 1 -- 172.21.15.26:0/1673899836 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f803c10f7f0 con 0x7f802006ec10 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.801+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 msgr2=0x7f80200710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.801+0000 7f8041e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 0x7f80200710c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f802c009b80 tx=0x7f802c00b040 comp rx=0 tx=0).stop 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.801+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 msgr2=0x7f803c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.801+0000 7f8041e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8024013870 tx=0x7f8024009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 shutdown_connections 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f802006ec10 0x7f80200710c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c0662f0 0x7f803c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 --2- 172.21.15.26:0/1673899836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f803c0fff70 0x7f803c10df10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 >> 172.21.15.26:0/1673899836 conn(0x7f803c0f74c0 msgr2=0x7f803c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:28.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 shutdown_connections 2024-04-16T07:31:28.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.802+0000 7f8041e00700 1 -- 172.21.15.26:0/1673899836 wait complete. 2024-04-16T07:31:28.817 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:31:28.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.959+0000 7f5c0e000700 1 -- 172.21.15.26:0/1633399281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c08071ea0 msgr2=0x7f5c080703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.959+0000 7f5c0e000700 1 --2- 172.21.15.26:0/1633399281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c08071ea0 0x7f5c080703b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8004940 tx=0x7f5bf8009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:28.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 -- 172.21.15.26:0/1633399281 shutdown_connections 2024-04-16T07:31:28.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 --2- 172.21.15.26:0/1633399281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c080708f0 0x7f5c08070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 --2- 172.21.15.26:0/1633399281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c08071ea0 0x7f5c080703b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 -- 172.21.15.26:0/1633399281 >> 172.21.15.26:0/1633399281 conn(0x7f5c080f8530 msgr2=0x7f5c080fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:28.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 -- 172.21.15.26:0/1633399281 shutdown_connections 2024-04-16T07:31:28.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.960+0000 7f5c0e000700 1 -- 172.21.15.26:0/1633399281 wait complete. 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 Processor -- start 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 -- start start 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c08071ea0 0x7f5c08114500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08114b00 con 0x7f5c080708f0 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.961+0000 7f5c0e000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08115350 con 0x7f5c08071ea0 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38948/0 (socket says 172.21.15.26:38948) 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 -- 172.21.15.26:0/2940456698 learned_addr learned my addr 172.21.15.26:0/2940456698 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:28.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c06a00700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c08071ea0 0x7f5c08114500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 -- 172.21.15.26:0/2940456698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c08071ea0 msgr2=0x7f5c08114500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:28.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c08071ea0 0x7f5c08114500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 -- 172.21.15.26:0/2940456698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf8005040 con 0x7f5c080708f0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.962+0000 7f5c07400700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8013870 tx=0x7f5bf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.963+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bf8021070 con 0x7f5c080708f0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.963+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bf801c400 con 0x7f5c080708f0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.963+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c081155d0 con 0x7f5c080708f0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.963+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bf800e070 con 0x7f5c080708f0 2024-04-16T07:31:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.963+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c08115ab0 con 0x7f5c080708f0 2024-04-16T07:31:28.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.965+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0805f5f0 con 0x7f5c080708f0 2024-04-16T07:31:28.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.965+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5bf801ca90 con 0x7f5c080708f0 2024-04-16T07:31:28.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.966+0000 7f5bff400700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 0x7f5bec0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:28.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.967+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f5bf808fbc0 con 0x7f5c080708f0 2024-04-16T07:31:28.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.967+0000 7f5c06a00700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 0x7f5bec0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:28.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.968+0000 7f5c06a00700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 0x7f5bec0710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4004600 tx=0x7f5bf400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:28.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:28.975+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5bf805c3f0 con 0x7f5c080708f0 2024-04-16T07:31:29.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:29.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:29.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:29.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:29.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:29.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:29.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:29.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.242+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c08115d90 con 0x7f5bec06ec10 2024-04-16T07:31:29.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.244+0000 7f5bff400700 1 -- 172.21.15.26:0/2940456698 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f5c08115d90 con 0x7f5bec06ec10 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 msgr2=0x7f5bec0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 0x7f5bec0710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4004600 tx=0x7f5bf400b040 comp rx=0 tx=0).stop 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 msgr2=0x7f5c08113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8013870 tx=0x7f5bf8009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 shutdown_connections 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5bec06ec10 0x7f5bec0710c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c080708f0 0x7f5c08113fc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.246+0000 7f5c0e000700 1 --2- 172.21.15.26:0/2940456698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c08071ea0 0x7f5c08114500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.247+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 >> 172.21.15.26:0/2940456698 conn(0x7f5c080f8530 msgr2=0x7f5c080f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:29.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.247+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 shutdown_connections 2024-04-16T07:31:29.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.247+0000 7f5c0e000700 1 -- 172.21.15.26:0/2940456698 wait complete. 2024-04-16T07:31:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 -- 172.21.15.26:0/1135048234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c073180 msgr2=0x7fd88c073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 --2- 172.21.15.26:0/1135048234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c073180 0x7fd88c073b40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd878004940 tx=0x7fd878009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 -- 172.21.15.26:0/1135048234 shutdown_connections 2024-04-16T07:31:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 --2- 172.21.15.26:0/1135048234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c073180 0x7fd88c073b40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 --2- 172.21.15.26:0/1135048234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c072830 0x7fd88c072c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 -- 172.21.15.26:0/1135048234 >> 172.21.15.26:0/1135048234 conn(0x7fd88c0fa5d0 msgr2=0x7fd88c0fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:29.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 -- 172.21.15.26:0/1135048234 shutdown_connections 2024-04-16T07:31:29.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.397+0000 7fd891e00700 1 -- 172.21.15.26:0/1135048234 wait complete. 2024-04-16T07:31:29.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 Processor -- start 2024-04-16T07:31:29.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 -- start start 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c073180 0x7fd88c114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1151d0 con 0x7fd88c072830 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.398+0000 7fd891e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c115340 con 0x7fd88c073180 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38968/0 (socket says 172.21.15.26:38968) 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 -- 172.21.15.26:0/2104531750 learned_addr learned my addr 172.21.15.26:0/2104531750 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:29.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd88be00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c073180 0x7fd88c114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 -- 172.21.15.26:0/2104531750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c073180 msgr2=0x7fd88c114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c073180 0x7fd88c114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 -- 172.21.15.26:0/2104531750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878005040 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.399+0000 7fd890a00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd880002a80 tx=0x7fd880004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.400+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd880007cc0 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.400+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd88000f040 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.400+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c1155c0 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.400+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd88000dad0 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.400+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c115b80 con 0x7fd88c072830 2024-04-16T07:31:29.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.401+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c05f5f0 con 0x7fd88c072830 2024-04-16T07:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.402+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd88000dc30 con 0x7fd88c072830 2024-04-16T07:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.403+0000 7fd888c00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 0x7fd8740710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.403+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fd88008e730 con 0x7fd88c072830 2024-04-16T07:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.403+0000 7fd88be00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 0x7fd8740710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.404+0000 7fd88be00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 0x7fd8740710c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd878009fd0 tx=0x7fd878009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.408+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd8800573b0 con 0x7fd88c072830 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:29.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:29.424+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:29.466 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:29 smithi104 ceph-mon[81027]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:29.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:29 smithi026 ceph-mon[92215]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.654+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd88c115df0 con 0x7fd87406ec10 2024-04-16T07:31:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.672+0000 7fd888c00700 1 -- 172.21.15.26:0/2104531750 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fd88c115df0 con 0x7fd87406ec10 2024-04-16T07:31:29.671 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:31:29.671 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (8m) 76s ago 18m 14.0M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (5m) 76s ago 19m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (5m) 23s ago 17m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (8m) 76s ago 18m 73.0M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (111s) 76s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (83s) 76s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (53s) 23s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (27s) 23s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 d300ba6bee79 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (8m) 76s ago 20m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:31:29.672 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (7m) 23s ago 17m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (6m) 76s ago 20m 49.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (6m) 23s ago 17m 39.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 starting - - - - 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 starting - - - - 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (4m) 76s ago 16m 51.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (4m) 76s ago 15m 51.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:31:29.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (3m) 76s ago 15m 43.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:31:29.674 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (3m) 23s ago 14m 52.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:31:29.674 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (2m) 23s ago 14m 42.3M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:31:29.674 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (2m) 23s ago 13m 42.9M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:31:29.674 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (7m) 76s ago 18m 60.1M - 2.43.0 a07b618ecd1d cbcd99128095 2024-04-16T07:31:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 msgr2=0x7fd8740710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 0x7fd8740710c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd878009fd0 tx=0x7fd878009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 msgr2=0x7fd88c113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd880002a80 tx=0x7fd880004600 comp rx=0 tx=0).stop 2024-04-16T07:31:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 shutdown_connections 2024-04-16T07:31:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd87406ec10 0x7fd8740710c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd88c072830 0x7fd88c113f20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 --2- 172.21.15.26:0/2104531750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88c073180 0x7fd88c114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 >> 172.21.15.26:0/2104531750 conn(0x7fd88c0fa5d0 msgr2=0x7fd88c0fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 shutdown_connections 2024-04-16T07:31:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.675+0000 7fd891e00700 1 -- 172.21.15.26:0/2104531750 wait complete. 2024-04-16T07:31:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 -- 172.21.15.26:0/3460362065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f81046d0 msgr2=0x7f56f8104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 --2- 172.21.15.26:0/3460362065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f81046d0 0x7f56f8104b40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f56e4004940 tx=0x7f56e4009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 -- 172.21.15.26:0/3460362065 shutdown_connections 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 --2- 172.21.15.26:0/3460362065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f81046d0 0x7f56f8104b40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 --2- 172.21.15.26:0/3460362065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f8103420 0x7f56f8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 -- 172.21.15.26:0/3460362065 >> 172.21.15.26:0/3460362065 conn(0x7f56f8072640 msgr2=0x7f56f8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 -- 172.21.15.26:0/3460362065 shutdown_connections 2024-04-16T07:31:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.826+0000 7f56fdc00700 1 -- 172.21.15.26:0/3460362065 wait complete. 2024-04-16T07:31:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 Processor -- start 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 -- start start 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f81046d0 0x7f56f8114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8114a90 con 0x7f56f8103420 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.827+0000 7f56fdc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8114c00 con 0x7f56f81046d0 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38978/0 (socket says 172.21.15.26:38978) 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 -- 172.21.15.26:0/2333345104 learned_addr learned my addr 172.21.15.26:0/2333345104 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6000700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f81046d0 0x7f56f8114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 -- 172.21.15.26:0/2333345104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f81046d0 msgr2=0x7f56f8114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f81046d0 0x7f56f8114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.828+0000 7f56f6a00700 1 -- 172.21.15.26:0/2333345104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e4005040 con 0x7f56f8103420 2024-04-16T07:31:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56f6a00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56e8002a50 tx=0x7f56e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e8007cc0 con 0x7f56f8103420 2024-04-16T07:31:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56e800f040 con 0x7f56f8103420 2024-04-16T07:31:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f8119660 con 0x7f56f8103420 2024-04-16T07:31:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e800da30 con 0x7f56f8103420 2024-04-16T07:31:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.829+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f8119c20 con 0x7f56f8103420 2024-04-16T07:31:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.830+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f804c710 con 0x7f56f8103420 2024-04-16T07:31:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.831+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f56e800db90 con 0x7f56f8103420 2024-04-16T07:31:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.832+0000 7f56eea00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 0x7f56dc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.832+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f56e808e700 con 0x7f56f8103420 2024-04-16T07:31:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.833+0000 7f56f6000700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 0x7f56dc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.834+0000 7f56f6000700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 0x7f56dc0710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f56e400b3b0 tx=0x7f56e4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:29.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:29.837+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f56e805aeb0 con 0x7f56f8103420 2024-04-16T07:31:30.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:30.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:30.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.194+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f56f811a150 con 0x7f56f8103420 2024-04-16T07:31:30.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.195+0000 7f56eea00700 1 -- 172.21.15.26:0/2333345104 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f56e805aa40 con 0x7f56f8103420 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:31:30.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:31:30.196 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:31:30.196 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T07:31:30.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:31:30.196 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:31:30.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.197+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 msgr2=0x7f56dc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.197+0000 7f56fdc00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 0x7f56dc0710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f56e400b3b0 tx=0x7f56e4009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.197+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 msgr2=0x7f56f8113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.197+0000 7f56fdc00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56e8002a50 tx=0x7f56e80046b0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 shutdown_connections 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f56dc06ec00 0x7f56dc0710b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56f8103420 0x7f56f8113f50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 --2- 172.21.15.26:0/2333345104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56f81046d0 0x7f56f8114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 >> 172.21.15.26:0/2333345104 conn(0x7f56f8072640 msgr2=0x7f56f8073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 shutdown_connections 2024-04-16T07:31:30.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.198+0000 7f56fdc00700 1 -- 172.21.15.26:0/2333345104 wait complete. 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 -- 172.21.15.26:0/3369485007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 msgr2=0x7f0544102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 --2- 172.21.15.26:0/3369485007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f0544102160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f052c004940 tx=0x7f052c009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 -- 172.21.15.26:0/3369485007 shutdown_connections 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 --2- 172.21.15.26:0/3369485007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f0544102160 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 --2- 172.21.15.26:0/3369485007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 0x7f05440ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 -- 172.21.15.26:0/3369485007 >> 172.21.15.26:0/3369485007 conn(0x7f05440f7270 msgr2=0x7f05440f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 -- 172.21.15.26:0/3369485007 shutdown_connections 2024-04-16T07:31:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.353+0000 7f0543200700 1 -- 172.21.15.26:0/3369485007 wait complete. 2024-04-16T07:31:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 Processor -- start 2024-04-16T07:31:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 -- start start 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 0x7f054410fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05441105d0 con 0x7f05440ffd40 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.354+0000 7f0543200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544110dd0 con 0x7f05440662f0 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38996/0 (socket says 172.21.15.26:38996) 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 -- 172.21.15.26:0/2268454696 learned_addr learned my addr 172.21.15.26:0/2268454696 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541e00700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 0x7f054410fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 -- 172.21.15.26:0/2268454696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 msgr2=0x7f054410fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 0x7f054410fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.355+0000 7f0541400700 1 -- 172.21.15.26:0/2268454696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c005040 con 0x7f05440ffd40 2024-04-16T07:31:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f0541400700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b0d0 tx=0x7f052c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f052c00e070 con 0x7f05440ffd40 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f052c00b410 con 0x7f05440ffd40 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f052c0125e0 con 0x7f05440ffd40 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0544111050 con 0x7f05440ffd40 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.356+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0544111540 con 0x7f05440ffd40 2024-04-16T07:31:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.357+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054405f5f0 con 0x7f05440ffd40 2024-04-16T07:31:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.358+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f052c00b580 con 0x7f05440ffd40 2024-04-16T07:31:30.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.359+0000 7f053a000700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 0x7f05280710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.359+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f052c08f690 con 0x7f05440ffd40 2024-04-16T07:31:30.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.359+0000 7f0541e00700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 0x7f05280710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.360+0000 7f0541e00700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 0x7f05280710c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0534009b80 tx=0x7f053400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.364+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f052c05bec0 con 0x7f05440ffd40 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:30.424+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:30.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:30 smithi026 ceph-mon[92215]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:30.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:30 smithi026 ceph-mon[92215]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:30.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:30 smithi026 ceph-mon[92215]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:30 smithi104 ceph-mon[81027]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:30 smithi104 ceph-mon[81027]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:30.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:30 smithi104 ceph-mon[81027]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:30.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.655+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0544105710 con 0x7f05440ffd40 2024-04-16T07:31:30.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.655+0000 7f053a000700 1 -- 172.21.15.26:0/2268454696 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1940 (secure 0 0 0) 0x7f052c0227b0 con 0x7f05440ffd40 2024-04-16T07:31:30.655 INFO:teuthology.orchestra.run.smithi026.stdout:e36 2024-04-16T07:31:30.655 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:31:30.655 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:31:30.655 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 35 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:56.852323+0000 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:31:30.656 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 85 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=34404} 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:31:30.657 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 34404 members: 34404 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:34404} state up:active seq 22 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:31:30.658 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:44307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 msgr2=0x7f05280710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 0x7f05280710c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0534009b80 tx=0x7f053400b040 comp rx=0 tx=0).stop 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 msgr2=0x7f054410ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b0d0 tx=0x7f052c009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 shutdown_connections 2024-04-16T07:31:30.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f052806ec10 0x7f05280710c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05440662f0 0x7f054410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 --2- 172.21.15.26:0/2268454696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05440ffd40 0x7f054410ffd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.658+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 >> 172.21.15.26:0/2268454696 conn(0x7f05440f7270 msgr2=0x7f05440fb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.659+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 shutdown_connections 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.659+0000 7f0543200700 1 -- 172.21.15.26:0/2268454696 wait complete. 2024-04-16T07:31:30.660 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 36 2024-04-16T07:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.813+0000 7fa98ce00700 1 -- 172.21.15.26:0/3016729251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 msgr2=0x7fa9880fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.813+0000 7fa98ce00700 1 --2- 172.21.15.26:0/3016729251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9880fe460 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa974004940 tx=0x7fa974009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.813+0000 7fa98ce00700 1 -- 172.21.15.26:0/3016729251 shutdown_connections 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.813+0000 7fa98ce00700 1 --2- 172.21.15.26:0/3016729251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9880fe460 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.813+0000 7fa98ce00700 1 --2- 172.21.15.26:0/3016729251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 0x7fa9880fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.814+0000 7fa98ce00700 1 -- 172.21.15.26:0/3016729251 >> 172.21.15.26:0/3016729251 conn(0x7fa9880f8580 msgr2=0x7fa9880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.814+0000 7fa98ce00700 1 -- 172.21.15.26:0/3016729251 shutdown_connections 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.814+0000 7fa98ce00700 1 -- 172.21.15.26:0/3016729251 wait complete. 2024-04-16T07:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.814+0000 7fa98ce00700 1 Processor -- start 2024-04-16T07:31:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa98ce00700 1 -- start start 2024-04-16T07:31:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa98ce00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 0x7fa98810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa98ce00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa98ce00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881106a0 con 0x7fa9880fdff0 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa98ce00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa988110810 con 0x7fa9880fce80 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa985400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa985400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39008/0 (socket says 172.21.15.26:39008) 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa985400700 1 -- 172.21.15.26:0/2327003248 learned_addr learned my addr 172.21.15.26:0/2327003248 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.815+0000 7fa985e00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 0x7fa98810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.816+0000 7fa985400700 1 -- 172.21.15.26:0/2327003248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 msgr2=0x7fa98810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.816+0000 7fa985400700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 0x7fa98810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.816+0000 7fa985400700 1 -- 172.21.15.26:0/2327003248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974005040 con 0x7fa9880fdff0 2024-04-16T07:31:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.816+0000 7fa985400700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b3b0 tx=0x7fa974009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.816+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa97400e040 con 0x7fa9880fdff0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.817+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa97400bbf0 con 0x7fa9880fdff0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.817+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa988115210 con 0x7fa9880fdff0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.817+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa97401c630 con 0x7fa9880fdff0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.817+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9881157d0 con 0x7fa9880fdff0 2024-04-16T07:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.818+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98804c710 con 0x7fa9880fdff0 2024-04-16T07:31:30.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.819+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa97400bd60 con 0x7fa9880fdff0 2024-04-16T07:31:30.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.820+0000 7fa97e000700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 0x7fa96c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:30.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.820+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa97408f540 con 0x7fa9880fdff0 2024-04-16T07:31:30.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.820+0000 7fa985e00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 0x7fa96c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:30.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.821+0000 7fa985e00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 0x7fa96c0710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa978004f40 tx=0x7fa97800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:30.825+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa97405bd70 con 0x7fa9880fdff0 2024-04-16T07:31:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.104+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa988102230 con 0x7fa96c06ec00 2024-04-16T07:31:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.106+0000 7fa97e000700 1 -- 172.21.15.26:0/2327003248 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7fa988102230 con 0x7fa96c06ec00 2024-04-16T07:31:31.105 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:31:31.105 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "mds", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:31:31.106 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 msgr2=0x7fa96c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 0x7fa96c0710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa978004f40 tx=0x7fa97800b040 comp rx=0 tx=0).stop 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 msgr2=0x7fa9881100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b3b0 tx=0x7fa974009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 shutdown_connections 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa96c06ec00 0x7fa96c0710b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9880fce80 0x7fa98810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 --2- 172.21.15.26:0/2327003248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9880fdff0 0x7fa9881100a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.108+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 >> 172.21.15.26:0/2327003248 conn(0x7fa9880f8580 msgr2=0x7fa9880f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.109+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 shutdown_connections 2024-04-16T07:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.109+0000 7fa98ce00700 1 -- 172.21.15.26:0/2327003248 wait complete. 2024-04-16T07:31:31.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:31.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:31.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:31.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:31.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:31.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:31.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 -- 172.21.15.26:0/126624069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fe0e0 msgr2=0x7f4c100fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 --2- 172.21.15.26:0/126624069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fe0e0 0x7f4c100fe550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc004940 tx=0x7f4bfc009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 -- 172.21.15.26:0/126624069 shutdown_connections 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 --2- 172.21.15.26:0/126624069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fe0e0 0x7f4c100fe550 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 --2- 172.21.15.26:0/126624069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fcef0 0x7f4c100fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 -- 172.21.15.26:0/126624069 >> 172.21.15.26:0/126624069 conn(0x7f4c100f8530 msgr2=0x7f4c100fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 -- 172.21.15.26:0/126624069 shutdown_connections 2024-04-16T07:31:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.259+0000 7f4c17e00700 1 -- 172.21.15.26:0/126624069 wait complete. 2024-04-16T07:31:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.260+0000 7f4c17e00700 1 Processor -- start 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.260+0000 7f4c17e00700 1 -- start start 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c17e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c17e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fe0e0 0x7f4c10110110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c17e00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10110710 con 0x7f4c100fcef0 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c17e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10110f60 con 0x7f4c100fe0e0 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39024/0 (socket says 172.21.15.26:39024) 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 -- 172.21.15.26:0/1452481727 learned_addr learned my addr 172.21.15.26:0/1452481727 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c14a00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fe0e0 0x7f4c10110110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 -- 172.21.15.26:0/1452481727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fe0e0 msgr2=0x7f4c10110110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fe0e0 0x7f4c10110110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.261+0000 7f4c15400700 1 -- 172.21.15.26:0/1452481727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc005040 con 0x7f4c100fcef0 2024-04-16T07:31:31.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c15400700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400ecb0 tx=0x7f4c0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:31.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c04007b30 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c0400f040 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c0400cca0 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c10111240 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.262+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c10111780 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.263+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c10101a70 con 0x7f4c100fcef0 2024-04-16T07:31:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.264+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4c04007c90 con 0x7f4c100fcef0 2024-04-16T07:31:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.266+0000 7f4c0d600700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 0x7f4bf80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:31:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.266+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f4c0408ea30 con 0x7f4c100fcef0 2024-04-16T07:31:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.266+0000 7f4c14a00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 0x7f4bf80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:31:31.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.267+0000 7f4c14a00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 0x7f4bf80710c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00b0d0 tx=0x7f4bfc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:31:31.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.271+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4c0405b290 con 0x7f4c100fcef0 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:31.425+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:31 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2333345104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:31 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:31 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:31 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2268454696' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:31:31.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:31 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2333345104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:31.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:31 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:31.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:31 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:31.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:31 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2268454696' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:31:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.640+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4c10105a00 con 0x7f4c100fcef0 2024-04-16T07:31:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.640+0000 7f4c0d600700 1 -- 172.21.15.26:0/1452481727 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f4c0405ae20 con 0x7f4c100fcef0 2024-04-16T07:31:31.640 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:31:31.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 msgr2=0x7f4bf80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 0x7f4bf80710c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00b0d0 tx=0x7f4bfc009e30 comp rx=0 tx=0).stop 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 msgr2=0x7f4c1010fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400ecb0 tx=0x7f4c0400b040 comp rx=0 tx=0).stop 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 shutdown_connections 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4bf806ec10 0x7f4bf80710c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c100fcef0 0x7f4c1010fbd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 --2- 172.21.15.26:0/1452481727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c100fe0e0 0x7f4c10110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:31:31.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.643+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 >> 172.21.15.26:0/1452481727 conn(0x7f4c100f8530 msgr2=0x7f4c100f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:31:31.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.644+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 shutdown_connections 2024-04-16T07:31:31.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:31:31.644+0000 7f4c17e00700 1 -- 172.21.15.26:0/1452481727 wait complete. 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:32.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:32.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:32.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:31:32.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:31:32.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 35) v5 -- 0x7fe9ac0060f0 con 0x7fe9b4014240 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.425+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.426+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 4 ==== client_session(renewcaps seq 35) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:31:32.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:32.426+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 35) v1 from mds.0 2024-04-16T07:31:32.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:32 smithi026 ceph-mon[92215]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:32.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:32 smithi026 ceph-mon[92215]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:32.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:32 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1452481727' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:31:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:32 smithi104 ceph-mon[81027]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:31:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:32 smithi104 ceph-mon[81027]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:32 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1452481727' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:33.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:33.138+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:33.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:33.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:33.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:33.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:33.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:33.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:33.425+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:34.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:33 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:33 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:34.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:34.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:34.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:34.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:34.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:34.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:34.139+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:34.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:34.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:34.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:34.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:34.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:34.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:34.425+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:35.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:34 smithi026 ceph-mon[92215]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:35.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:34 smithi104 ceph-mon[81027]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:35.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:35.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:35.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:35.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:35.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:35.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:35.139+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.425+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.425+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.425+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.426+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:35.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:35.426+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:36.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:36.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:36.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:36.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:36.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:36.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:36.139+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:36.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:36.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:36.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:36.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:36.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:36.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.426+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:36.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113305 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.695+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 646, success, outsize: 120 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.695+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.695+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.695+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.695+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113389 2024-04-16T07:31:36.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 648, success, outsize: 120 2024-04-16T07:31:36.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.710+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.710+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.710+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.710+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.710+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113476 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 650, success, outsize: 120 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.726+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.726+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.726+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.726+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.726+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113569 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 652, success, outsize: 120 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.743+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.743+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.743+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.743+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.743+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.760 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113659 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 654, success, outsize: 120 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.762+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.762+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.762+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.762+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.762+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.775 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113743 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 656, success, outsize: 120 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.777+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.777+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.777+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.777+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.777+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113830 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.793+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 658, success, outsize: 120 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.793+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.793+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.793+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.793+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113923 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 660, success, outsize: 120 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.809+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.809+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.809+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.809+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.809+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114013 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 662, success, outsize: 120 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.825+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.825+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.825+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.825+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.825+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114097 2024-04-16T07:31:36.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 664, success, outsize: 120 2024-04-16T07:31:36.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.839+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.839+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.839+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.839+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.839+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114184 2024-04-16T07:31:36.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.857+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 666, success, outsize: 120 2024-04-16T07:31:36.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.857+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.857+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.857+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.857+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114277 2024-04-16T07:31:36.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.877+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 668, success, outsize: 120 2024-04-16T07:31:36.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.877+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.877+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.877+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.877+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114367 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 670, success, outsize: 120 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.896+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.896+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.896+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.896+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.896+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114451 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 672, success, outsize: 120 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.914+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.914+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.914+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.914+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.914+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114538 2024-04-16T07:31:36.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.932+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 674, success, outsize: 120 2024-04-16T07:31:36.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.932+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.932+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.932+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.932+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114631 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 676, success, outsize: 120 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.947+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.947+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.947+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.947+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.947+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114721 2024-04-16T07:31:36.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.963+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 678, success, outsize: 120 2024-04-16T07:31:36.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.963+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.963+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.963+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.963+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114806 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 680, success, outsize: 120 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.977+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.977+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.977+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.977+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.977+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114894 2024-04-16T07:31:36.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 682, success, outsize: 120 2024-04-16T07:31:36.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.990+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:36.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.990+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:36.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.990+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:36.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.990+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:36.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:36.990+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114988 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 684, success, outsize: 120 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.004+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.004+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.004+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.004+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.004+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115078 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 686, success, outsize: 120 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.018+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.018+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.018+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.018+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.018+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115162 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 688, success, outsize: 120 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.032+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.032+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.032+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.032+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.032+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115249 2024-04-16T07:31:37.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 690, success, outsize: 120 2024-04-16T07:31:37.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.045+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.045+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.045+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.045+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.045+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115342 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 692, success, outsize: 120 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.059+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.059+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.059+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.059+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.059+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.067 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:36 smithi026 ceph-mon[92215]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:37.067 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:37.067 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:36 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:37.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115432 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.079+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 694, success, outsize: 120 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.079+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.079+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.079+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.079+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:36 smithi104 ceph-mon[81027]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:37.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:36 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:37.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115516 2024-04-16T07:31:37.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 696, success, outsize: 120 2024-04-16T07:31:37.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.100+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.100+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.100+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.100+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.100+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115603 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 698, success, outsize: 120 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.113+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.113+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.113+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.113+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.113+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115697 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr: unique: 700, success, outsize: 120 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.126+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.126+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.126+0000 7fe9d8cc2300 10 client.24339 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:37.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.126+0000 7fe9d8cc2300 3 client.24339 ll_getattr 0x1.head = 0 2024-04-16T07:31:37.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.126+0000 7fe9d8cc2300 8 client.24339 _ll_forget 0x1 1 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:37.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:37.139+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:37.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:37.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:37.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:37.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:37.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:37.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:37.426+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:38.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:38.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:38.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:38.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:38.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:38.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:38.140+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:38.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:38.427+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:39.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:39.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:39.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:39.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 236 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:39.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:39.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.140+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:38 smithi026 ceph-mon[92215]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:38 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:39.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:38 smithi104 ceph-mon[81027]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:39.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:39.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:38 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:39.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:39.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:39.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:39.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:39.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:39.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:39.427+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:39.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128196 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 478, success, outsize: 120 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.961+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.961+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.961+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.961+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:39.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.961+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128278 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 480, success, outsize: 120 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.981+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.981+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.981+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.981+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:39.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:39.981+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128363 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 482, success, outsize: 120 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.001+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.001+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.001+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.001+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.001+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128454 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 484, success, outsize: 120 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.024+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.024+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.024+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.024+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.024+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128542 2024-04-16T07:31:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.050+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 486, success, outsize: 120 2024-04-16T07:31:40.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.050+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.050+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.050+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.050+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128624 2024-04-16T07:31:40.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 488, success, outsize: 120 2024-04-16T07:31:40.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.072+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.072+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.072+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.072+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.072+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128709 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 490, success, outsize: 120 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.097+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.097+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.097+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.097+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.097+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128800 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 492, success, outsize: 120 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.121+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.121+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.121+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.121+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.121+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:40.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:40.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:40.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 244 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:40.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:40.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.140+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128888 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 494, success, outsize: 120 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.145+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.145+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.145+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.145+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.145+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128970 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 496, success, outsize: 120 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.166+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.166+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.166+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.166+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.166+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129055 2024-04-16T07:31:40.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 498, success, outsize: 120 2024-04-16T07:31:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.189+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.189+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.189+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.189+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.189+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129146 2024-04-16T07:31:40.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 500, success, outsize: 120 2024-04-16T07:31:40.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.213+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.213+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.213+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.213+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.213+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129234 2024-04-16T07:31:40.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 502, success, outsize: 120 2024-04-16T07:31:40.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.237+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.237+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.237+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.237+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.237+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129316 2024-04-16T07:31:40.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 504, success, outsize: 120 2024-04-16T07:31:40.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.261+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.261+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.261+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.261+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.261+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129401 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 506, success, outsize: 120 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.284+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.284+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.284+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.284+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.284+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129492 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 508, success, outsize: 120 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.308+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.308+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.308+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.308+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.308+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129580 2024-04-16T07:31:40.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 510, success, outsize: 120 2024-04-16T07:31:40.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.331+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.331+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.331+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.331+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.331+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129662 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 512, success, outsize: 120 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.353+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.353+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.353+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.353+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.353+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129747 2024-04-16T07:31:40.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 514, success, outsize: 120 2024-04-16T07:31:40.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.377+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.377+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.377+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.377+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.377+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129838 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 516, success, outsize: 120 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.401+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.401+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.401+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.401+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.401+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129926 2024-04-16T07:31:40.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 518, success, outsize: 120 2024-04-16T07:31:40.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.425+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.425+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.425+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.425+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.425+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:40.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:40.427+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:40.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130008 2024-04-16T07:31:40.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 520, success, outsize: 120 2024-04-16T07:31:40.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.446+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.446+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.446+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.446+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.446+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130093 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.466+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 522, success, outsize: 120 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.466+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.466+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.466+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.466+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130184 2024-04-16T07:31:40.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 524, success, outsize: 120 2024-04-16T07:31:40.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.489+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.489+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.489+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.489+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.489+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130272 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 526, success, outsize: 120 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.514+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.514+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.514+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.514+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.514+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130354 2024-04-16T07:31:40.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 528, success, outsize: 120 2024-04-16T07:31:40.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.536+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.536+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.536+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.536+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.536+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130439 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.561+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 530, success, outsize: 120 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.561+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.561+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.561+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.561+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:40.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130530 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 532, success, outsize: 120 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.584+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.584+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.584+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.584+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:31:40.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:40.584+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:41.140+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:41 smithi026 ceph-mon[92215]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:41 smithi104 ceph-mon[81027]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:41.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:41.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:41.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:41.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:41.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:41.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:41.427+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:42.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.140+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:42.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.140+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:42.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.140+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:42.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.140+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:42.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.141+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:42.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:42.141+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:42.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:42.428+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:43.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:42 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:43.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:42 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:31:43.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:43.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:43.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:43.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:43.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:43.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:43.141+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:43.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:43.428+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 36) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:44.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.141+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:44.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.142+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 5 ==== client_session(renewcaps seq 36) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:31:44.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:44.142+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 36) v1 from mds.0 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:44.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:44.428+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:44.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:44 smithi026 ceph-mon[92215]: pgmap v245: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:45.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:44 smithi104 ceph-mon[81027]: pgmap v245: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:45.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:45.141+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:45.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:45.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:45.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:45.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:45.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:45.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:45.429+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:45 smithi026 ceph-mon[92215]: Upgrade: Updating prometheus.smithi026 2024-04-16T07:31:46.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:45 smithi026 ceph-mon[92215]: Deploying daemon prometheus.smithi026 on smithi026 2024-04-16T07:31:46.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:45 smithi104 ceph-mon[81027]: Upgrade: Updating prometheus.smithi026 2024-04-16T07:31:46.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:45 smithi104 ceph-mon[81027]: Deploying daemon prometheus.smithi026 on smithi026 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:46.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:46.141+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:46.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:46.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:46.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:46.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:46.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:46.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:46.428+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:46.979 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:46 smithi026 ceph-mon[92215]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:47.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:46 smithi104 ceph-mon[81027]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:47.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:47.142+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:47.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:47.429+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:48.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:48.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:48.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:48.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:48.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:48.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:48.142+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:48.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:48.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:48.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:48.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:48.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:48.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:48.429+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:48 smithi026 ceph-mon[92215]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:48 smithi104 ceph-mon[81027]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:49.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:49.142+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:49.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:49.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:49.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:49.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:49.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:49.430+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:50.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:50.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:31:50.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:50.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:50.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:50.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:50.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:50.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:50.142+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:50.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:50.430+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:51.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:50 smithi026 ceph-mon[92215]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:50 smithi104 ceph-mon[81027]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:51.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:51.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:51.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:51.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:51.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:51.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:51.142+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:51.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:51.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:51.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:51.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:51.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:51.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:51.430+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:52.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:52.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:52.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:52.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:52.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:52.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:52.143+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:52.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:52.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:31:52.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:31:52.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 36) v5 -- 0x7fe9ac0064e0 con 0x7fe9b4014240 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.430+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.431+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 5 ==== client_session(renewcaps seq 36) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:31:52.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:52.431+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 36) v1 from mds.0 2024-04-16T07:31:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:52 smithi026 ceph-mon[92215]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:53.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:52 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:53.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:52 smithi104 ceph-mon[81027]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:52 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:31:53.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:53.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:53.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:53.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:53.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:53.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:53.143+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:53.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:53.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:53.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:53.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:53.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:53.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:53.431+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:54.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:54.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:54.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:54.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:54.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:54.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:54.143+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.430+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.430+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.430+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.431+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:54.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:54.431+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:55.027 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:54 smithi026 ceph-mon[92215]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:54 smithi104 ceph-mon[81027]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:55.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:55.143+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:55.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:55.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:55.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:55.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:55.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:55.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:55.431+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:56.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:56.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:56.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:56.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:56.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:56.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:56.144+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:56.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:56.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:56.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:56.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:56.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:56.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:56.432+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:57.015 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:56 smithi026 ceph-mon[92215]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:56 smithi104 ceph-mon[81027]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:57.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:57.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:57.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:57.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:57.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:57.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:57.144+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:57.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:57.432+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:58.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:58.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:58.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:58.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:58.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:58.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:58.144+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.431+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.432+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.432+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:58.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:58.432+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:58 smithi104 ceph-mon[81027]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:31:58 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:31:59.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:31:59.144+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:31:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:58 smithi026 ceph-mon[92215]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:31:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:31:58 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:31:59.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:31:59.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:31:59.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:31:59.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:31:59.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:31:59.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:31:59.432+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.144+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.144+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.144+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.144+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.145+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:00.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:00.145+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:00.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:00.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:00.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:00.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:00.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:00.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:00.432+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:00 smithi104 ceph-mon[81027]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:01.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:01.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:01.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:01.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:01.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:01.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:01.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:01.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:01.145+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:00 smithi026 ceph-mon[92215]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:01.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:01.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:01.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:01.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:01.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:01.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:01.433+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:01.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 -- 172.21.15.26:0/3308494014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c1046d0 msgr2=0x7ff95c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 --2- 172.21.15.26:0/3308494014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c1046d0 0x7ff95c104b40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff94c004940 tx=0x7ff94c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 -- 172.21.15.26:0/3308494014 shutdown_connections 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 --2- 172.21.15.26:0/3308494014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c1046d0 0x7ff95c104b40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 --2- 172.21.15.26:0/3308494014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c103420 0x7ff95c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 -- 172.21.15.26:0/3308494014 >> 172.21.15.26:0/3308494014 conn(0x7ff95c072640 msgr2=0x7ff95c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 -- 172.21.15.26:0/3308494014 shutdown_connections 2024-04-16T07:32:01.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.816+0000 7ff964800700 1 -- 172.21.15.26:0/3308494014 wait complete. 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.817+0000 7ff964800700 1 Processor -- start 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.817+0000 7ff964800700 1 -- start start 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.817+0000 7ff964800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff964800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c1046d0 0x7ff95c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff964800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c114b20 con 0x7ff95c103420 2024-04-16T07:32:01.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff964800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c1194c0 con 0x7ff95c1046d0 2024-04-16T07:32:01.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:01.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43136/0 (socket says 172.21.15.26:43136) 2024-04-16T07:32:01.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 -- 172.21.15.26:0/203428859 learned_addr learned my addr 172.21.15.26:0/203428859 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 -- 172.21.15.26:0/203428859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c1046d0 msgr2=0x7ff95c114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961400700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c1046d0 0x7ff95c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c1046d0 0x7ff95c114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.818+0000 7ff961e00700 1 -- 172.21.15.26:0/203428859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c005040 con 0x7ff95c103420 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff961e00700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b790 tx=0x7ff950009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff95000f770 con 0x7ff95c103420 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff95000fda0 con 0x7ff95c103420 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff95000d310 con 0x7ff95c103420 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c119720 con 0x7ff95c103420 2024-04-16T07:32:01.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.819+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c119c90 con 0x7ff95c103420 2024-04-16T07:32:01.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.820+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c04c710 con 0x7ff95c103420 2024-04-16T07:32:01.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.821+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff950010420 con 0x7ff95c103420 2024-04-16T07:32:01.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.822+0000 7ff956000700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 0x7ff9440710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:01.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.823+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7ff95008e9a0 con 0x7ff95c103420 2024-04-16T07:32:01.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.823+0000 7ff961400700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 0x7ff9440710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:01.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.824+0000 7ff961400700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 0x7ff9440710b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00b970 tx=0x7ff94c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:01.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:01.828+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff95005b1d0 con 0x7ff95c103420 2024-04-16T07:32:02.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.096+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff95c108970 con 0x7ff94406ec00 2024-04-16T07:32:02.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.100+0000 7ff956000700 1 -- 172.21.15.26:0/203428859 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7ff95c108970 con 0x7ff94406ec00 2024-04-16T07:32:02.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 msgr2=0x7ff9440710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 0x7ff9440710b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00b970 tx=0x7ff94c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 msgr2=0x7ff95c113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b790 tx=0x7ff950009250 comp rx=0 tx=0).stop 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 shutdown_connections 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff94406ec00 0x7ff9440710b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff95c103420 0x7ff95c113f50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 --2- 172.21.15.26:0/203428859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c1046d0 0x7ff95c114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 >> 172.21.15.26:0/203428859 conn(0x7ff95c072640 msgr2=0x7ff95c073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 shutdown_connections 2024-04-16T07:32:02.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.103+0000 7ff964800700 1 -- 172.21.15.26:0/203428859 wait complete. 2024-04-16T07:32:02.119 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:32:02.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:02.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:02.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:02.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:02.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:02.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:02.145+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3786857096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4103420 msgr2=0x7f5eb4103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3786857096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4103420 0x7f5eb4103830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4004940 tx=0x7f5ea4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3786857096 shutdown_connections 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3786857096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 0x7f5eb4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3786857096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4103420 0x7f5eb4103830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3786857096 >> 172.21.15.26:0/3786857096 conn(0x7f5eb4072640 msgr2=0x7f5eb4074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3786857096 shutdown_connections 2024-04-16T07:32:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.267+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3786857096 wait complete. 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.268+0000 7f5ebaa00700 1 Processor -- start 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5ebaa00700 1 -- start start 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5ebaa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 0x7f5eb4114100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5ebaa00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5ebaa00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4114b00 con 0x7f5eb4114640 2024-04-16T07:32:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5ebaa00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4114c40 con 0x7f5eb41046d0 2024-04-16T07:32:02.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5eb3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5eb3400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43154/0 (socket says 172.21.15.26:43154) 2024-04-16T07:32:02.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.269+0000 7f5eb3400700 1 -- 172.21.15.26:0/3952208758 learned_addr learned my addr 172.21.15.26:0/3952208758 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:02.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eb3e00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 0x7f5eb4114100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eb3400700 1 -- 172.21.15.26:0/3952208758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 msgr2=0x7f5eb4114100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eb3400700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 0x7f5eb4114100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eb3400700 1 -- 172.21.15.26:0/3952208758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4005040 con 0x7f5eb4114640 2024-04-16T07:32:02.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eb3400700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c830 tx=0x7f5ea0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:02.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea000a760 con 0x7f5eb4114640 2024-04-16T07:32:02.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.270+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb4119c10 con 0x7f5eb4114640 2024-04-16T07:32:02.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.271+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea000ad90 con 0x7f5eb4114640 2024-04-16T07:32:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.271+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea00113a0 con 0x7f5eb4114640 2024-04-16T07:32:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.271+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb411a180 con 0x7f5eb4114640 2024-04-16T07:32:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.272+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb404c710 con 0x7f5eb4114640 2024-04-16T07:32:02.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.274+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ea0012400 con 0x7f5eb4114640 2024-04-16T07:32:02.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.275+0000 7f5eabe00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 0x7f5e980710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.275+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f5ea008ea60 con 0x7f5eb4114640 2024-04-16T07:32:02.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.275+0000 7f5eb3e00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 0x7f5e980710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.276+0000 7f5eb3e00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 0x7f5e980710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b970 tx=0x7f5ea4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:02.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.280+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5ea005b210 con 0x7f5eb4114640 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:02.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:02.433+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:02.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.539+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5eb4108bc0 con 0x7f5e9806ec00 2024-04-16T07:32:02.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.542+0000 7f5eabe00700 1 -- 172.21.15.26:0/3952208758 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7f5eb4108bc0 con 0x7f5e9806ec00 2024-04-16T07:32:02.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.545+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 msgr2=0x7f5e980710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.545+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 0x7f5e980710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b970 tx=0x7f5ea4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.545+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 msgr2=0x7f5eb4119670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.545+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c830 tx=0x7f5ea0004470 comp rx=0 tx=0).stop 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 shutdown_connections 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f5e9806ec00 0x7f5e980710b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5eb41046d0 0x7f5eb4114100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 --2- 172.21.15.26:0/3952208758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5eb4114640 0x7f5eb4119670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 >> 172.21.15.26:0/3952208758 conn(0x7f5eb4072640 msgr2=0x7f5eb4073e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:02.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 shutdown_connections 2024-04-16T07:32:02.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.546+0000 7f5ebaa00700 1 -- 172.21.15.26:0/3952208758 wait complete. 2024-04-16T07:32:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 -- 172.21.15.26:0/1629100652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 msgr2=0x7f4a080fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1629100652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a080fe070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49f4004940 tx=0x7f49f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 -- 172.21.15.26:0/1629100652 shutdown_connections 2024-04-16T07:32:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1629100652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a080fe070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1629100652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 0x7f4a080fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 -- 172.21.15.26:0/1629100652 >> 172.21.15.26:0/1629100652 conn(0x7f4a080f8540 msgr2=0x7f4a080fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 -- 172.21.15.26:0/1629100652 shutdown_connections 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.702+0000 7f4a0f200700 1 -- 172.21.15.26:0/1629100652 wait complete. 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 Processor -- start 2024-04-16T07:32:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 -- start start 2024-04-16T07:32:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 0x7f4a0810fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08110890 con 0x7f4a080fd710 2024-04-16T07:32:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.703+0000 7f4a0f200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08110fd0 con 0x7f4a080fcdc0 2024-04-16T07:32:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43168/0 (socket says 172.21.15.26:43168) 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 -- 172.21.15.26:0/1083902206 learned_addr learned my addr 172.21.15.26:0/1083902206 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07e00700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 0x7f4a0810fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 -- 172.21.15.26:0/1083902206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 msgr2=0x7f4a0810fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 0x7f4a0810fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.704+0000 7f4a07400700 1 -- 172.21.15.26:0/1083902206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4005040 con 0x7f4a080fd710 2024-04-16T07:32:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f4a07400700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b970 tx=0x7f49f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49f400e040 con 0x7f4a080fd710 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49f4004290 con 0x7f4a080fd710 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a08111250 con 0x7f4a080fd710 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49f401c630 con 0x7f4a080fd710 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.705+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a08111810 con 0x7f4a080fd710 2024-04-16T07:32:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.706+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08101a60 con 0x7f4a080fd710 2024-04-16T07:32:02.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.707+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f49f400bbf0 con 0x7f4a080fd710 2024-04-16T07:32:02.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.708+0000 7f49ffe00700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 0x7f49ec0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:02.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.708+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f49f408fdd0 con 0x7f4a080fd710 2024-04-16T07:32:02.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.709+0000 7f4a07e00700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 0x7f49ec0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:02.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.710+0000 7f4a07e00700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 0x7f49ec0710c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007fd0 tx=0x7f49f8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:02.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.714+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f49f405c580 con 0x7f4a080fd710 2024-04-16T07:32:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.958+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a08105aa0 con 0x7f49ec06ec10 2024-04-16T07:32:02.958 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:02 smithi026 ceph-mon[92215]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.986+0000 7f49ffe00700 1 -- 172.21.15.26:0/1083902206 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f4a08105aa0 con 0x7f49ec06ec10 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (8m) 4s ago 19m 14.1M - 0.25.0 c8568f914cd2 a69147c099d9 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (5m) 4s ago 19m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (5m) 32s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (8m) 4s ago 18m 71.7M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (2m) 4s ago 12m 17.7M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:32:02.986 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (116s) 4s ago 12m 11.5M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (87s) 32s ago 12m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (61s) 32s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 d300ba6bee79 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (8m) 4s ago 21m 513M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (7m) 32s ago 18m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (7m) 4s ago 21m 54.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (6m) 32s ago 17m 39.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (43s) 4s ago 19m 7688k - 1.5.0 0da6a335fe13 2562d9ea91f2 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (36s) 32s ago 18m 6689k - 1.5.0 0da6a335fe13 8fdb5a9c24bb 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (5m) 4s ago 17m 52.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (4m) 4s ago 16m 52.8M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (4m) 4s ago 15m 43.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (3m) 32s ago 15m 52.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (3m) 32s ago 14m 42.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:32:02.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (2m) 32s ago 14m 43.0M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:32:02.988 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (11s) 4s ago 18m 40.5M - 2.43.0 a07b618ecd1d b4054ff04ef2 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 msgr2=0x7f49ec0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 0x7f49ec0710c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007fd0 tx=0x7f49f8009040 comp rx=0 tx=0).stop 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 msgr2=0x7f4a08110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b970 tx=0x7f49f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 shutdown_connections 2024-04-16T07:32:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f49ec06ec10 0x7f49ec0710c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a080fcdc0 0x7f4a0810fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 --2- 172.21.15.26:0/1083902206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a080fd710 0x7f4a08110290 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 >> 172.21.15.26:0/1083902206 conn(0x7f4a080f8540 msgr2=0x7f4a080f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.989+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 shutdown_connections 2024-04-16T07:32:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:02.990+0000 7f4a0f200700 1 -- 172.21.15.26:0/1083902206 wait complete. 2024-04-16T07:32:03.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:02 smithi104 ceph-mon[81027]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 -- 172.21.15.26:0/2478229135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 msgr2=0x7f4474101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 --2- 172.21.15.26:0/2478229135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f4474101e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4464004940 tx=0x7f4464009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:03.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:03.145+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 -- 172.21.15.26:0/2478229135 shutdown_connections 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 --2- 172.21.15.26:0/2478229135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4474102370 0x7f4474104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 --2- 172.21.15.26:0/2478229135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f4474101e30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 -- 172.21.15.26:0/2478229135 >> 172.21.15.26:0/2478229135 conn(0x7f44740f74c0 msgr2=0x7f44740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 -- 172.21.15.26:0/2478229135 shutdown_connections 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.145+0000 7f4479800700 1 -- 172.21.15.26:0/2478229135 wait complete. 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.146+0000 7f4479800700 1 Processor -- start 2024-04-16T07:32:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4479800700 1 -- start start 2024-04-16T07:32:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4479800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4479800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4474102370 0x7f44741101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4479800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4474110860 con 0x7f44740662f0 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4479800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4474111000 con 0x7f4474102370 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4472a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4472a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43176/0 (socket says 172.21.15.26:43176) 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.147+0000 7f4472a00700 1 -- 172.21.15.26:0/3202245081 learned_addr learned my addr 172.21.15.26:0/3202245081 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.148+0000 7f4472a00700 1 -- 172.21.15.26:0/3202245081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4474102370 msgr2=0x7f44741101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.148+0000 7f4472a00700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4474102370 0x7f44741101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.148+0000 7f4472a00700 1 -- 172.21.15.26:0/3202245081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4464005040 con 0x7f44740662f0 2024-04-16T07:32:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.148+0000 7f4472a00700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f446400b970 tx=0x7f4464009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.149+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446400e040 con 0x7f44740662f0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.149+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4464004290 con 0x7f44740662f0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.149+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446401c630 con 0x7f44740662f0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.149+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4474111280 con 0x7f44740662f0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.149+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4474111840 con 0x7f44740662f0 2024-04-16T07:32:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.150+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447405f5f0 con 0x7f44740662f0 2024-04-16T07:32:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.151+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f446400bbf0 con 0x7f44740662f0 2024-04-16T07:32:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.152+0000 7f446aa00700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 0x7f44580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.152+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f446408fcd0 con 0x7f44740662f0 2024-04-16T07:32:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.153+0000 7f446be00700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 0x7f44580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.153+0000 7f446be00700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 0x7f44580710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f445c006fd0 tx=0x7f445c006db0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.157+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f446405c500 con 0x7f44740662f0 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:03.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:03.433+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:03.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.495+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4474111b70 con 0x7f44740662f0 2024-04-16T07:32:03.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.496+0000 7f446aa00700 1 -- 172.21.15.26:0/3202245081 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4464013e60 con 0x7f44740662f0 2024-04-16T07:32:03.495 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:32:03.495 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T07:32:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:32:03.497 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:32:03.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 msgr2=0x7f44580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 0x7f44580710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f445c006fd0 tx=0x7f445c006db0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 msgr2=0x7f447410fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f446400b970 tx=0x7f4464009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:03.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 shutdown_connections 2024-04-16T07:32:03.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f445806ec10 0x7f44580710c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44740662f0 0x7f447410fc90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 --2- 172.21.15.26:0/3202245081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4474102370 0x7f44741101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.499+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 >> 172.21.15.26:0/3202245081 conn(0x7f44740f74c0 msgr2=0x7f44740f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:03.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.500+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 shutdown_connections 2024-04-16T07:32:03.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.500+0000 7f4479800700 1 -- 172.21.15.26:0/3202245081 wait complete. 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 -- 172.21.15.26:0/513640431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00662f0 msgr2=0x7f0de00ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 --2- 172.21.15.26:0/513640431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00662f0 0x7f0de00ffa30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009ce0 tx=0x7f0dd0004d10 comp rx=0 tx=0).stop 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 -- 172.21.15.26:0/513640431 shutdown_connections 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 --2- 172.21.15.26:0/513640431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00fff70 0x7f0de0102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 --2- 172.21.15.26:0/513640431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00662f0 0x7f0de00ffa30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 -- 172.21.15.26:0/513640431 >> 172.21.15.26:0/513640431 conn(0x7f0de00f74c0 msgr2=0x7f0de00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 -- 172.21.15.26:0/513640431 shutdown_connections 2024-04-16T07:32:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.655+0000 7f0de5800700 1 -- 172.21.15.26:0/513640431 wait complete. 2024-04-16T07:32:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.656+0000 7f0de5800700 1 Processor -- start 2024-04-16T07:32:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.656+0000 7f0de5800700 1 -- start start 2024-04-16T07:32:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0de5800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00662f0 0x7f0de010fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0de5800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0de5800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0110a30 con 0x7f0de00fff70 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0de5800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0111170 con 0x7f0de00662f0 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0dde000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0dde000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43202/0 (socket says 172.21.15.26:43202) 2024-04-16T07:32:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.657+0000 7f0dde000700 1 -- 172.21.15.26:0/1197657728 learned_addr learned my addr 172.21.15.26:0/1197657728 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:03.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0ddea00700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00662f0 0x7f0de010fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:03.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0dde000700 1 -- 172.21.15.26:0/1197657728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00662f0 msgr2=0x7f0de010fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0dde000700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00662f0 0x7f0de010fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0dde000700 1 -- 172.21.15.26:0/1197657728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd0009940 con 0x7f0de00fff70 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0dde000700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00c810 tx=0x7f0dcc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.658+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dcc012810 con 0x7f0de00fff70 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.659+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dcc012e40 con 0x7f0de00fff70 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.659+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dcc00a370 con 0x7f0de00fff70 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.659+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de0111450 con 0x7f0de00fff70 2024-04-16T07:32:03.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.659+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de0111a10 con 0x7f0de00fff70 2024-04-16T07:32:03.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.660+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de00f8ff0 con 0x7f0de00fff70 2024-04-16T07:32:03.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.661+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0dcc011810 con 0x7f0de00fff70 2024-04-16T07:32:03.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.662+0000 7f0dd6a00700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 0x7f0dc40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:03.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.662+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f0dcc08edd0 con 0x7f0de00fff70 2024-04-16T07:32:03.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.662+0000 7f0ddea00700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 0x7f0dc40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:03.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.663+0000 7f0ddea00700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 0x7f0dc40710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0015040 tx=0x7f0dd0004d10 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:03.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.667+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0dcc05b580 con 0x7f0de00fff70 2024-04-16T07:32:03.946 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:03 smithi026 ceph-mon[92215]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:03.946 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:03 smithi026 ceph-mon[92215]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:03.946 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:03 smithi026 ceph-mon[92215]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:03.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:03 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3202245081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:03.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.946+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0de01115e0 con 0x7f0de00fff70 2024-04-16T07:32:03.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.947+0000 7f0dd6a00700 1 -- 172.21.15.26:0/1197657728 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1940 (secure 0 0 0) 0x7f0dcc0167b0 con 0x7f0de00fff70 2024-04-16T07:32:03.947 INFO:teuthology.orchestra.run.smithi026.stdout:e36 2024-04-16T07:32:03.947 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 35 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:56.852323+0000 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:32:03.948 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 85 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=34404} 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:32:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 34404 members: 34404 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:34404} state up:active seq 22 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:44307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 msgr2=0x7f0dc40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 0x7f0dc40710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0015040 tx=0x7f0dd0004d10 comp rx=0 tx=0).stop 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 msgr2=0x7f0de0110430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00c810 tx=0x7f0dcc004470 comp rx=0 tx=0).stop 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 shutdown_connections 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0dc406ec10 0x7f0dc40710c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de00662f0 0x7f0de010fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 --2- 172.21.15.26:0/1197657728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de00fff70 0x7f0de0110430 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:03.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.951+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 >> 172.21.15.26:0/1197657728 conn(0x7f0de00f74c0 msgr2=0x7f0de00fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:03.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.952+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 shutdown_connections 2024-04-16T07:32:03.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:03.952+0000 7f0de5800700 1 -- 172.21.15.26:0/1197657728 wait complete. 2024-04-16T07:32:03.952 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 36 2024-04-16T07:32:04.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:03 smithi104 ceph-mon[81027]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:04.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:03 smithi104 ceph-mon[81027]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:04.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:03 smithi104 ceph-mon[81027]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:04.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:03 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3202245081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 -- 172.21.15.26:0/1758277469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 msgr2=0x7ff6a80fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 --2- 172.21.15.26:0/1758277469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a80fe550 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff694004940 tx=0x7ff694009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 -- 172.21.15.26:0/1758277469 shutdown_connections 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 --2- 172.21.15.26:0/1758277469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a80fe550 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 --2- 172.21.15.26:0/1758277469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6a80fcef0 0x7ff6a80fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.103+0000 7ff6ae200700 1 -- 172.21.15.26:0/1758277469 >> 172.21.15.26:0/1758277469 conn(0x7ff6a80f8530 msgr2=0x7ff6a80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.104+0000 7ff6ae200700 1 -- 172.21.15.26:0/1758277469 shutdown_connections 2024-04-16T07:32:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.104+0000 7ff6ae200700 1 -- 172.21.15.26:0/1758277469 wait complete. 2024-04-16T07:32:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.104+0000 7ff6ae200700 1 Processor -- start 2024-04-16T07:32:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.104+0000 7ff6ae200700 1 -- start start 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6ae200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6a80fcef0 0x7ff6a810fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6ae200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6ae200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8110f80 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6ae200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a81110f0 con 0x7ff6a80fcef0 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43228/0 (socket says 172.21.15.26:43228) 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 -- 172.21.15.26:0/4066341976 learned_addr learned my addr 172.21.15.26:0/4066341976 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 -- 172.21.15.26:0/4066341976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6a80fcef0 msgr2=0x7ff6a810fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6a80fcef0 0x7ff6a810fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.105+0000 7ff6a6a00700 1 -- 172.21.15.26:0/4066341976 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694005040 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff6a6a00700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff694009150 tx=0x7ff694009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69400e040 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff69400bbf0 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69401c630 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a8111370 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.106+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a8111930 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.107+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a8101a70 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.109+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff69400bd60 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.109+0000 7ff69f400700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 0x7ff68c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.110+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7ff69408fd80 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.110+0000 7ff6a7400700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 0x7ff68c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.110+0000 7ff6a7400700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 0x7ff68c0710c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff698006fd0 tx=0x7ff698006db0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.115+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff69405c5b0 con 0x7ff6a80fe0e0 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 37) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:04.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:04.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 264 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:04.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:04.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.145+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:04.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.146+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 6 ==== client_session(renewcaps seq 37) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:32:04.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.146+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 37) v1 from mds.0 2024-04-16T07:32:04.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.363+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6a8111ba0 con 0x7ff68c06ec10 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.368+0000 7ff69f400700 1 -- 172.21.15.26:0/4066341976 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7ff6a8111ba0 con 0x7ff68c06ec10 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "mds", 2024-04-16T07:32:04.368 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:32:04.369 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:32:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 msgr2=0x7ff68c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 0x7ff68c0710c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff698006fd0 tx=0x7ff698006db0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 msgr2=0x7ff6a8110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff694009150 tx=0x7ff694009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 shutdown_connections 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff68c06ec10 0x7ff68c0710c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6a80fcef0 0x7ff6a810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 --2- 172.21.15.26:0/4066341976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6a80fe0e0 0x7ff6a8110270 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 >> 172.21.15.26:0/4066341976 conn(0x7ff6a80f8530 msgr2=0x7ff6a80f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 shutdown_connections 2024-04-16T07:32:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.372+0000 7ff6ae200700 1 -- 172.21.15.26:0/4066341976 wait complete. 2024-04-16T07:32:04.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:04.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:04.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:04.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:04.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:04.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:04.433+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 -- 172.21.15.26:0/2960238227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fcef0 msgr2=0x7f397c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 --2- 172.21.15.26:0/2960238227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fcef0 0x7f397c0fd320 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3970004940 tx=0x7f3970009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 -- 172.21.15.26:0/2960238227 shutdown_connections 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 --2- 172.21.15.26:0/2960238227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fe0e0 0x7f397c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 --2- 172.21.15.26:0/2960238227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fcef0 0x7f397c0fd320 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 -- 172.21.15.26:0/2960238227 >> 172.21.15.26:0/2960238227 conn(0x7f397c0f8530 msgr2=0x7f397c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 -- 172.21.15.26:0/2960238227 shutdown_connections 2024-04-16T07:32:04.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.517+0000 7f3984000700 1 -- 172.21.15.26:0/2960238227 wait complete. 2024-04-16T07:32:04.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.518+0000 7f3984000700 1 Processor -- start 2024-04-16T07:32:04.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.518+0000 7f3984000700 1 -- start start 2024-04-16T07:32:04.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3984000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fcef0 0x7f397c10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3984000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3984000700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c110820 con 0x7f397c0fe0e0 2024-04-16T07:32:04.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3984000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c111010 con 0x7f397c0fcef0 2024-04-16T07:32:04.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3980c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:04.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3980c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43254/0 (socket says 172.21.15.26:43254) 2024-04-16T07:32:04.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3980c00700 1 -- 172.21.15.26:0/1183944723 learned_addr learned my addr 172.21.15.26:0/1183944723 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3981600700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fcef0 0x7f397c10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.519+0000 7f3980c00700 1 -- 172.21.15.26:0/1183944723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fcef0 msgr2=0x7f397c10fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3980c00700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fcef0 0x7f397c10fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3980c00700 1 -- 172.21.15.26:0/1183944723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970005040 con 0x7f397c0fe0e0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3980c00700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f396800b790 tx=0x7f3968009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f396800f770 con 0x7f397c0fe0e0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c1112f0 con 0x7f397c0fe0e0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f396800fda0 con 0x7f397c0fe0e0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f396800d310 con 0x7f397c0fe0e0 2024-04-16T07:32:04.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.520+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c1118b0 con 0x7f397c0fe0e0 2024-04-16T07:32:04.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.522+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c101a70 con 0x7f397c0fe0e0 2024-04-16T07:32:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.523+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3968010420 con 0x7f397c0fe0e0 2024-04-16T07:32:04.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.525+0000 7f3979600700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 0x7f39640710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:04.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.525+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f396808e5e0 con 0x7f397c0fe0e0 2024-04-16T07:32:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.525+0000 7f3981600700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 0x7f39640710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.526+0000 7f3981600700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 0x7f39640710c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f397000b970 tx=0x7f3970009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:04.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.529+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f396805ae30 con 0x7f397c0fe0e0 2024-04-16T07:32:04.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132350 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 534, success, outsize: 120 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.598+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.598+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.598+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.598+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.598+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132438 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 536, success, outsize: 120 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.614+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.614+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.614+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.614+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.614+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.628 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132529 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 538, success, outsize: 120 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.629+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.629+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.629+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.629+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.629+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132626 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 540, success, outsize: 120 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.647+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.647+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.647+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.647+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.647+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.664 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132720 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 542, success, outsize: 120 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.665+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.665+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.665+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.665+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.665+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132808 2024-04-16T07:32:04.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 544, success, outsize: 120 2024-04-16T07:32:04.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.682+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.682+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.682+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.682+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.682+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132899 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 546, success, outsize: 120 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.700+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.700+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.700+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.700+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.700+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132996 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 548, success, outsize: 120 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.719+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.719+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.719+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.719+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.719+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133090 2024-04-16T07:32:04.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.737+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 550, success, outsize: 120 2024-04-16T07:32:04.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.737+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.737+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.737+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.737+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.754 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133178 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.756+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 552, success, outsize: 120 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.756+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.756+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.756+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.756+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133269 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 554, success, outsize: 120 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.775+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.775+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.775+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.775+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.775+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133366 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 556, success, outsize: 120 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.791+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.791+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.791+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.791+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.791+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133460 2024-04-16T07:32:04.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 558, success, outsize: 120 2024-04-16T07:32:04.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.811+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.811+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.811+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.811+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.811+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133548 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 560, success, outsize: 120 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.829+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.829+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.829+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.829+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.829+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133639 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 562, success, outsize: 120 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.851+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.851+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.851+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.851+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.851+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.867+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f397c111b30 con 0x7f397c0fe0e0 2024-04-16T07:32:04.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.867+0000 7f3979600700 1 -- 172.21.15.26:0/1183944723 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f396805a9c0 con 0x7f397c0fe0e0 2024-04-16T07:32:04.867 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133737 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 564, success, outsize: 120 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.868+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.868+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.868+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.868+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.868+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 msgr2=0x7f39640710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 0x7f39640710c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f397000b970 tx=0x7f3970009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 msgr2=0x7f397c110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f396800b790 tx=0x7f3968009250 comp rx=0 tx=0).stop 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 shutdown_connections 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f396406ec10 0x7f39640710c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0fcef0 0x7f397c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 --2- 172.21.15.26:0/1183944723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f397c0fe0e0 0x7f397c110190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 >> 172.21.15.26:0/1183944723 conn(0x7f397c0f8530 msgr2=0x7f397c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 shutdown_connections 2024-04-16T07:32:04.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:04.870+0000 7f3984000700 1 -- 172.21.15.26:0/1183944723 wait complete. 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133832 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 566, success, outsize: 120 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.887+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.887+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.887+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.887+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.887+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133921 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 568, success, outsize: 120 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.901+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.901+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.901+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.901+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.901+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134012 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 570, success, outsize: 120 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.916+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.916+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.916+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.916+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.916 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.916+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134109 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 572, success, outsize: 120 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.932+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.932+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.932+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.932+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.932+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134203 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 574, success, outsize: 120 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.950+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.950+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.950+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.950+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.950+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134291 2024-04-16T07:32:04.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 576, success, outsize: 120 2024-04-16T07:32:04.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.964+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.964+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.964+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.964+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.964+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134382 2024-04-16T07:32:04.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 578, success, outsize: 120 2024-04-16T07:32:04.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.979+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.979+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.979+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.979+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.979+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:04.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134479 2024-04-16T07:32:04.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 580, success, outsize: 120 2024-04-16T07:32:04.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.995+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:04.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.995+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:04.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.995+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:04.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.995+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:04.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:04.995+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:05.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134573 2024-04-16T07:32:05.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 582, success, outsize: 120 2024-04-16T07:32:05.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.011+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:05.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.011+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:05.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.011+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.011+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:05.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.011+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:05.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134661 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 584, success, outsize: 120 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.025+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.025+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.025+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.025+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:05.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.025+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134752 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 586, success, outsize: 120 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.042+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.042+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.042+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:05.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.042+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:05.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.042+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134849 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 588, success, outsize: 120 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.060+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.060+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.060+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.060+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:05.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.060+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:05.066 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:04 smithi026 ceph-mon[92215]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:05.066 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:04 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1197657728' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:32:05.066 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:04 smithi104 ceph-mon[81027]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:04 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1197657728' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:32:05.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.145+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.145+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.145+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.145+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.146+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:05.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:05.146+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:05.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:05.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:05.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:05.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:05.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:05.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:05.434+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:05 smithi104 ceph-mon[81027]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:05 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1183944723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:32:06.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:06.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:06.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:06.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:06.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:06.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:06.146+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:06.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:05 smithi026 ceph-mon[92215]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:06.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:05 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1183944723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:06.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:06.434+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:07.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:07.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:07.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:07.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:07.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:07.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:07.146+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:32:07.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.336 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.336 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:07 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:07 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:07.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:07.434+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:08.147+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:08.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:08 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:32:08.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:08.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:08.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:08.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:08.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:08.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:08.435+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:08.437 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:08 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T07:32:09.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:09.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:09.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:09.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:09.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:09.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:09.146+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:09.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:09 smithi104 ceph-mon[81027]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:09.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:09.434+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:09.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:09 smithi026 ceph-mon[92215]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:10.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:10.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:10.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:10.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:10.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:10.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:10.147+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:10.227 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:10 smithi026 ceph-mon[92215]: Upgrade: Updating alertmanager.smithi026 2024-04-16T07:32:10.228 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:10 smithi026 ceph-mon[92215]: Deploying daemon alertmanager.smithi026 on smithi026 2024-04-16T07:32:10.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:10 smithi104 ceph-mon[81027]: Upgrade: Updating alertmanager.smithi026 2024-04-16T07:32:10.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:10 smithi104 ceph-mon[81027]: Deploying daemon alertmanager.smithi026 on smithi026 2024-04-16T07:32:10.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:10.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:10.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:10.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:10.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:10.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:10.435+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:11.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:11.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:11.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:11.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:11.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:11.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:11.147+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:11.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:11 smithi026 ceph-mon[92215]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:11.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:11 smithi104 ceph-mon[81027]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:11.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:11.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:11.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:11.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:11.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:11.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:11.436+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:12.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:12.148+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 37) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.435+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:12.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.436+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 6 ==== client_session(renewcaps seq 37) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:12.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:12.436+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 37) v1 from mds.0 2024-04-16T07:32:13.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.147+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:13.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.147+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:13.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.147+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:13.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.147+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:13.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.148+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:13.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:13.148+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:13.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:13 smithi104 ceph-mon[81027]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:13.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:13.436+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:13.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:13 smithi026 ceph-mon[92215]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:14.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:14.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:14.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:14.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:14.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:14.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:14.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:14.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:14.436+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:15.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:15.147+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:15.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:15 smithi026 ceph-mon[92215]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:15.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:15 smithi104 ceph-mon[81027]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:15.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:15.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.148+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.148+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.148+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.148+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:16.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:16.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:16.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:16.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:17.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:17.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:17.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:17.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:17.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:17.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:17.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:17.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:17 smithi104 ceph-mon[81027]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:17.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:17.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:17.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:17.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:17.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:17.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:17.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:17.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:17.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:17.436+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:17 smithi026 ceph-mon[92215]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:18.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:18.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:18.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:18.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:18.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:18.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:18 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:18.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:18 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:18.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:18 smithi026 ceph-mon[92215]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:18.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:18 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:18.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:18 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:18.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:18 smithi104 ceph-mon[81027]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:19.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:19.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:19.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:19.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:19.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:19.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:19.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:19.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:19.436+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:19.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:19.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:20.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:20.149+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:20.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:20.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:20.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:20.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:20.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:20.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:20.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:20.666 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:20 smithi026 ceph-mon[92215]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:20.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:20 smithi104 ceph-mon[81027]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:21.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:21.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:21.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:21.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:21.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:21.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:21.150+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:21.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:21.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:22.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:22.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:22.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:22.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:22.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:22.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:22.150+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:22.437+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:22 smithi026 ceph-mon[92215]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:22.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:22 smithi104 ceph-mon[81027]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:23.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.078+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c0045c0 con 0x7fbf70008390 2024-04-16T07:32:23.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.078+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c0045c0 con 0x7fbf70008390 2024-04-16T07:32:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:23.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:23.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:23.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:23.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:23.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:23.150+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:23.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:23.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:23.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:23.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:23.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:23.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:23.438+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 38) v5 -- 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:32:24.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.150+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.151+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 7 ==== client_session(renewcaps seq 38) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:32:24.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:24.151+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 38) v1 from mds.0 2024-04-16T07:32:24.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:24.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:24.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:24.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:24.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:24.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:24.438+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:24 smithi104 ceph-mon[81027]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:24.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:24.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:24 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:25.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:24 smithi026 ceph-mon[92215]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:24 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:25.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.150+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:25.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.150+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:25.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.150+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:25.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.150+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:25.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.151+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:25.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:25.151+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:25.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:25.438+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:26.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:26.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:26.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:26.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:26.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:26.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:26.151+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:26.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:26.438+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:26.765 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:26 smithi026 ceph-mon[92215]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:26.765 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:26 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:26.765 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:26 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:26 smithi104 ceph-mon[81027]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:26 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:26 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:27.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:27.151+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:27.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:27.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:27.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:27.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:27.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:27.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:27.439+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:28.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:28.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:28.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:28.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:28.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:28.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:28.151+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:28.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:28.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:28.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:28.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:28.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:28.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:28.439+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:29.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:28 smithi026 ceph-mon[92215]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:29.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:28 smithi104 ceph-mon[81027]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:29.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:29.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 292 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:29.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.152+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:29.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:29.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:29.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:29.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:29.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:29.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:29.439+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:29.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136569 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 590, success, outsize: 120 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.900+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.900+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.900+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.900+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:29.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.900+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136651 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 592, success, outsize: 120 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.922+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.922+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.922+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.922+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:29.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.922+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136736 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 594, success, outsize: 120 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.945+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.945+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.945+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.945+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:29.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.945+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:29.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136827 2024-04-16T07:32:29.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 596, success, outsize: 120 2024-04-16T07:32:29.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.971+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:29.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.971+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:29.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.971+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:29.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.971+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:29.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.971+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:29.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136915 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 598, success, outsize: 120 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.998+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.998+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.998+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.998+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:29.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:29.998+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136997 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 600, success, outsize: 120 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.022+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.022+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.022+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.022+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.022+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137082 2024-04-16T07:32:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 602, success, outsize: 120 2024-04-16T07:32:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.043+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.043+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.043+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.043+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.043+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137173 2024-04-16T07:32:30.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 604, success, outsize: 120 2024-04-16T07:32:30.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.068+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.068+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.068+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.068+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.068+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137261 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 606, success, outsize: 120 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.093+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.093+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.093+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.093+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.093+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137343 2024-04-16T07:32:30.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 608, success, outsize: 120 2024-04-16T07:32:30.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.116+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.116+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.116+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.116+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.116+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137428 2024-04-16T07:32:30.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 610, success, outsize: 120 2024-04-16T07:32:30.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.140+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.140+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.140+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.140+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.140+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 303 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:30.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.152+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137519 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 612, success, outsize: 120 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.166+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.166+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.166+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.166+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.166+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137607 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 614, success, outsize: 120 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.191+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.191+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.191+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.191+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.191+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137689 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 616, success, outsize: 120 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.215+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.215+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.215+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.215+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.215+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137774 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 618, success, outsize: 120 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.241+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.241+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.241+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.241+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.241+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137865 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 620, success, outsize: 120 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.266+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.266+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.266+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.266+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.266+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137953 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 622, success, outsize: 120 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.289+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.289+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.289+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.289+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.289+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138035 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 624, success, outsize: 120 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.309+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.309+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.309+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.309+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.309+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138120 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 626, success, outsize: 120 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.336+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.337+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.337+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.337+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.337+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138211 2024-04-16T07:32:30.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 628, success, outsize: 120 2024-04-16T07:32:30.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.360+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.360+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.360+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.360+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.360+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138299 2024-04-16T07:32:30.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 630, success, outsize: 120 2024-04-16T07:32:30.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.384+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.384+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.384+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.384+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.384+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138381 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 632, success, outsize: 120 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.405+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.405+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.405+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.405+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.405+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138466 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 634, success, outsize: 120 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.424+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.424+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.424+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.424+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.424+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:30.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:30.440+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:30.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138557 2024-04-16T07:32:30.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 636, success, outsize: 120 2024-04-16T07:32:30.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.448+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.448+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.448+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.448+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.448+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138645 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 638, success, outsize: 120 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.472+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.472+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.472+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.472+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.472+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138727 2024-04-16T07:32:30.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 640, success, outsize: 120 2024-04-16T07:32:30.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.495+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.495+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.495+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.495+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.495+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138812 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 642, success, outsize: 120 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.516+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.516+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.516+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.516+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.516+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138903 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 644, success, outsize: 120 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.540+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.540+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.540+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.540+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:32:30.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:30.540+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:32:30.961 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:30 smithi026 ceph-mon[92215]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:30 smithi104 ceph-mon[81027]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:31.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:31.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:31.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:31.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:31.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:31.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:31.152+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:31.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.365+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004570 con 0x55fc3f5009a0 2024-04-16T07:32:31.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.366+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a0004570 con 0x55fc3f5009a0 2024-04-16T07:32:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:31.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:31.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:31.440+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:32.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:32.152+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:32.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 38) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.440+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:32.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.441+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 7 ==== client_session(renewcaps seq 38) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:32.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:32.441+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 38) v1 from mds.0 2024-04-16T07:32:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:32:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.763 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:32 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:32.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:32 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:33.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:33.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:33.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:33.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:33.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:33.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:33.152+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:33.440+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:34.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:34.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:34.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:34.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:34.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:34.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:34.153+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:34.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:34.441+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:34 smithi026 ceph-mon[92215]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:34 smithi104 ceph-mon[81027]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:35.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.039+0000 7fe108400700 1 -- 172.21.15.26:0/53830021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100071e20 msgr2=0x7fe100070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.039+0000 7fe108400700 1 --2- 172.21.15.26:0/53830021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100071e20 0x7fe100070330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4004940 tx=0x7fe0f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 -- 172.21.15.26:0/53830021 shutdown_connections 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 --2- 172.21.15.26:0/53830021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100070870 0x7fe100070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 --2- 172.21.15.26:0/53830021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100071e20 0x7fe100070330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 -- 172.21.15.26:0/53830021 >> 172.21.15.26:0/53830021 conn(0x7fe1000f8530 msgr2=0x7fe1000fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 -- 172.21.15.26:0/53830021 shutdown_connections 2024-04-16T07:32:35.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.040+0000 7fe108400700 1 -- 172.21.15.26:0/53830021 wait complete. 2024-04-16T07:32:35.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.041+0000 7fe108400700 1 Processor -- start 2024-04-16T07:32:35.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.041+0000 7fe108400700 1 -- start start 2024-04-16T07:32:35.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe108400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe108400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100071e20 0x7fe100110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe108400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100110860 con 0x7fe100070870 2024-04-16T07:32:35.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe108400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100110fa0 con 0x7fe100071e20 2024-04-16T07:32:35.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe105a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe105a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48466/0 (socket says 172.21.15.26:48466) 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.042+0000 7fe105a00700 1 -- 172.21.15.26:0/1301085367 learned_addr learned my addr 172.21.15.26:0/1301085367 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe105000700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100071e20 0x7fe100110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe105a00700 1 -- 172.21.15.26:0/1301085367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100071e20 msgr2=0x7fe100110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe105a00700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100071e20 0x7fe100110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe105a00700 1 -- 172.21.15.26:0/1301085367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4005040 con 0x7fe100070870 2024-04-16T07:32:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe105a00700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400b970 tx=0x7fe0f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.043+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f400e040 con 0x7fe100070870 2024-04-16T07:32:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.044+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe100111220 con 0x7fe100070870 2024-04-16T07:32:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.044+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0f4004290 con 0x7fe100070870 2024-04-16T07:32:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.044+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f401c630 con 0x7fe100070870 2024-04-16T07:32:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.044+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1001117e0 con 0x7fe100070870 2024-04-16T07:32:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.045+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10005f5f0 con 0x7fe100070870 2024-04-16T07:32:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.047+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe0f400bbf0 con 0x7fe100070870 2024-04-16T07:32:35.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.048+0000 7fe0fd600700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 0x7fe0e80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.048+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fe0f408fcd0 con 0x7fe100070870 2024-04-16T07:32:35.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.049+0000 7fe105000700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 0x7fe0e80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.050+0000 7fe105000700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 0x7fe0e80710c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec007fd0 tx=0x7fe0ec009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.059+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe0f405c500 con 0x7fe100070870 2024-04-16T07:32:35.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:35.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:35.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:35.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:35.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:35.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:35.153+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.321+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe100111c20 con 0x7fe0e806ec10 2024-04-16T07:32:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.322+0000 7fe0fd600700 1 -- 172.21.15.26:0/1301085367 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7fe100111c20 con 0x7fe0e806ec10 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 msgr2=0x7fe0e80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 0x7fe0e80710c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec007fd0 tx=0x7fe0ec009040 comp rx=0 tx=0).stop 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 msgr2=0x7fe10010fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400b970 tx=0x7fe0f4009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 shutdown_connections 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fe0e806ec10 0x7fe0e80710c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe100070870 0x7fe10010fd20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 --2- 172.21.15.26:0/1301085367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe100071e20 0x7fe100110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 >> 172.21.15.26:0/1301085367 conn(0x7fe1000f8530 msgr2=0x7fe1000f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 shutdown_connections 2024-04-16T07:32:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.325+0000 7fe108400700 1 -- 172.21.15.26:0/1301085367 wait complete. 2024-04-16T07:32:35.340 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:32:35.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:35.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:35.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:35.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:35.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:35.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:35.441+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 -- 172.21.15.26:0/1792732007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 msgr2=0x7f0e980fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 --2- 172.21.15.26:0/1792732007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e980fe460 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0e80004940 tx=0x7f0e80009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 -- 172.21.15.26:0/1792732007 shutdown_connections 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 --2- 172.21.15.26:0/1792732007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e980fe460 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 --2- 172.21.15.26:0/1792732007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 0x7f0e980fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 -- 172.21.15.26:0/1792732007 >> 172.21.15.26:0/1792732007 conn(0x7f0e980f8580 msgr2=0x7f0e980fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 -- 172.21.15.26:0/1792732007 shutdown_connections 2024-04-16T07:32:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.489+0000 7f0e9da00700 1 -- 172.21.15.26:0/1792732007 wait complete. 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 Processor -- start 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 -- start start 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 0x7f0e9810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98110680 con 0x7f0e980fdff0 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.490+0000 7f0e9da00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981107f0 con 0x7f0e980fce80 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48472/0 (socket says 172.21.15.26:48472) 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 -- 172.21.15.26:0/531229307 learned_addr learned my addr 172.21.15.26:0/531229307 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e96a00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 0x7f0e9810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 -- 172.21.15.26:0/531229307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 msgr2=0x7f0e9810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 0x7f0e9810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.491+0000 7f0e8e000700 1 -- 172.21.15.26:0/531229307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e80005040 con 0x7f0e980fdff0 2024-04-16T07:32:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.492+0000 7f0e8e000700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009150 tx=0x7f0e80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.494+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8000e040 con 0x7f0e980fdff0 2024-04-16T07:32:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.494+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8000bcc0 con 0x7f0e980fdff0 2024-04-16T07:32:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.494+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e981151f0 con 0x7f0e980fdff0 2024-04-16T07:32:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.494+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8001c630 con 0x7f0e980fdff0 2024-04-16T07:32:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.494+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e981157b0 con 0x7f0e980fdff0 2024-04-16T07:32:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.495+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98101980 con 0x7f0e980fdff0 2024-04-16T07:32:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.496+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0e80013420 con 0x7f0e980fdff0 2024-04-16T07:32:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.497+0000 7f0e8f400700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 0x7f0e7c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.497+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f0e8008f620 con 0x7f0e980fdff0 2024-04-16T07:32:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.497+0000 7f0e96a00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 0x7f0e7c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.498+0000 7f0e96a00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 0x7f0e7c0710b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e88004550 tx=0x7f0e88009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.502+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0e8005be50 con 0x7f0e980fdff0 2024-04-16T07:32:35.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.758+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e9802b2f0 con 0x7f0e7c06ec00 2024-04-16T07:32:35.758 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:35 smithi026 ceph-mon[92215]: Upgrade: Updating grafana.smithi026 2024-04-16T07:32:35.758 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:35 smithi026 ceph-mon[92215]: Deploying daemon grafana.smithi026 on smithi026 2024-04-16T07:32:35.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.760+0000 7f0e8f400700 1 -- 172.21.15.26:0/531229307 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f0e9802b2f0 con 0x7f0e7c06ec00 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 msgr2=0x7f0e7c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 0x7f0e7c0710b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e88004550 tx=0x7f0e88009040 comp rx=0 tx=0).stop 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 msgr2=0x7f0e98110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009150 tx=0x7f0e80009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 shutdown_connections 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0e7c06ec00 0x7f0e7c0710b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e980fce80 0x7f0e9810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 --2- 172.21.15.26:0/531229307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e980fdff0 0x7f0e98110080 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.763+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 >> 172.21.15.26:0/531229307 conn(0x7f0e980f8580 msgr2=0x7f0e980f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:35.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.764+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 shutdown_connections 2024-04-16T07:32:35.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.764+0000 7f0e9da00700 1 -- 172.21.15.26:0/531229307 wait complete. 2024-04-16T07:32:35.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:35 smithi104 ceph-mon[81027]: Upgrade: Updating grafana.smithi026 2024-04-16T07:32:35.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:35 smithi104 ceph-mon[81027]: Deploying daemon grafana.smithi026 on smithi026 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 -- 172.21.15.26:0/2246421453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c071f50 msgr2=0x7f634c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 --2- 172.21.15.26:0/2246421453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c071f50 0x7f634c070460 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f633c004940 tx=0x7f633c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 -- 172.21.15.26:0/2246421453 shutdown_connections 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 --2- 172.21.15.26:0/2246421453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f634c0709a0 0x7f634c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 --2- 172.21.15.26:0/2246421453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c071f50 0x7f634c070460 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 -- 172.21.15.26:0/2246421453 >> 172.21.15.26:0/2246421453 conn(0x7f634c0f8530 msgr2=0x7f634c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 -- 172.21.15.26:0/2246421453 shutdown_connections 2024-04-16T07:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.911+0000 7f6353a00700 1 -- 172.21.15.26:0/2246421453 wait complete. 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.912+0000 7f6353a00700 1 Processor -- start 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.912+0000 7f6353a00700 1 -- start start 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6353a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6353a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f634c071f50 0x7f634c1145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6353a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c114bd0 con 0x7f634c0709a0 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6353a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c115420 con 0x7f634c071f50 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48500/0 (socket says 172.21.15.26:48500) 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 -- 172.21.15.26:0/4251887551 learned_addr learned my addr 172.21.15.26:0/4251887551 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 -- 172.21.15.26:0/4251887551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f634c071f50 msgr2=0x7f634c1145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f634c071f50 0x7f634c1145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:35.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.913+0000 7f6351000700 1 -- 172.21.15.26:0/4251887551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c005040 con 0x7f634c0709a0 2024-04-16T07:32:35.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6351000700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f633c00b3b0 tx=0x7f633c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f633c00e040 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f633c00bbf0 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c1156a0 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f633c01c630 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.914+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c115be0 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.915+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c05f5f0 con 0x7f634c0709a0 2024-04-16T07:32:35.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.916+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f633c00bd60 con 0x7f634c0709a0 2024-04-16T07:32:35.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.918+0000 7f6349600700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 0x7f63300710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:35.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.918+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f633c08fd10 con 0x7f634c0709a0 2024-04-16T07:32:35.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.918+0000 7f6343e00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 0x7f63300710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:35.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.919+0000 7f6343e00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 0x7f63300710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6334004ad0 tx=0x7f6334007d40 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:35.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:35.923+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f633c05c540 con 0x7f634c0709a0 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:36.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:36.153+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.164+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f634c116020 con 0x7f633006ec10 2024-04-16T07:32:36.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.191+0000 7f6349600700 1 -- 172.21.15.26:0/4251887551 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f634c116020 con 0x7f633006ec10 2024-04-16T07:32:36.190 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (20s) 12s ago 19m 13.5M - 0.25.0 c8568f914cd2 2e8fc2d1047d 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (6m) 12s ago 20m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (6m) 65s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (9m) 12s ago 19m 72.7M - 9.4.7 954c08fa6188 4ecc2120b8f3 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (2m) 12s ago 12m 19.3M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (2m) 12s ago 12m 11.7M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (2m) 65s ago 12m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (94s) 65s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 d300ba6bee79 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (9m) 12s ago 21m 513M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (8m) 65s ago 18m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (7m) 12s ago 21m 55.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (7m) 65s ago 18m 39.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:32:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (76s) 12s ago 20m 7260k - 1.5.0 0da6a335fe13 2562d9ea91f2 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (69s) 65s ago 18m 6689k - 1.5.0 0da6a335fe13 8fdb5a9c24bb 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (5m) 12s ago 17m 53.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (5m) 12s ago 16m 52.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (4m) 12s ago 16m 44.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (4m) 65s ago 15m 52.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (3m) 65s ago 15m 42.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (3m) 65s ago 14m 43.0M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:32:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (44s) 12s ago 19m 48.3M - 2.43.0 a07b618ecd1d b4054ff04ef2 2024-04-16T07:32:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 msgr2=0x7f63300710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 0x7f63300710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6334004ad0 tx=0x7f6334007d40 comp rx=0 tx=0).stop 2024-04-16T07:32:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 msgr2=0x7f634c114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f633c00b3b0 tx=0x7f633c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 shutdown_connections 2024-04-16T07:32:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f633006ec10 0x7f63300710c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f634c0709a0 0x7f634c114090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 --2- 172.21.15.26:0/4251887551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f634c071f50 0x7f634c1145d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 >> 172.21.15.26:0/4251887551 conn(0x7f634c0f8530 msgr2=0x7f634c0f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.194+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 shutdown_connections 2024-04-16T07:32:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.195+0000 7f6353a00700 1 -- 172.21.15.26:0/4251887551 wait complete. 2024-04-16T07:32:36.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.354+0000 7ff5ffe00700 1 -- 172.21.15.26:0/3005575907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 msgr2=0x7ff6000fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.354+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/3005575907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff6000fe350 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec004940 tx=0x7ff5ec009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 -- 172.21.15.26:0/3005575907 shutdown_connections 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/3005575907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff6000fe350 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/3005575907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff6000fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 -- 172.21.15.26:0/3005575907 >> 172.21.15.26:0/3005575907 conn(0x7ff6000f83d0 msgr2=0x7ff6000fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 -- 172.21.15.26:0/3005575907 shutdown_connections 2024-04-16T07:32:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 -- 172.21.15.26:0/3005575907 wait complete. 2024-04-16T07:32:36.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.355+0000 7ff5ffe00700 1 Processor -- start 2024-04-16T07:32:36.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5ffe00700 1 -- start start 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5ffe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff600071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5ffe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5ffe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6001114d0 con 0x7ff6000fdee0 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5ffe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff600111640 con 0x7ff6000fcc30 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5fe000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5fe000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48514/0 (socket says 172.21.15.26:48514) 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.356+0000 7ff5fe000700 1 -- 172.21.15.26:0/1544059505 learned_addr learned my addr 172.21.15.26:0/1544059505 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fe000700 1 -- 172.21.15.26:0/1544059505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 msgr2=0x7ff600071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fea00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff600071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fe000700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff600071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fe000700 1 -- 172.21.15.26:0/1544059505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec005040 con 0x7ff6000fdee0 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fea00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff600071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:32:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5fe000700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00b3b0 tx=0x7ff5ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ec00e040 con 0x7ff6000fdee0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5ec004290 con 0x7ff6000fdee0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ec01c630 con 0x7ff6000fdee0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.357+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff600070910 con 0x7ff6000fdee0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.358+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff600070e80 con 0x7ff6000fdee0 2024-04-16T07:32:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.359+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff600105f30 con 0x7ff6000fdee0 2024-04-16T07:32:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.360+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff5ec00bbf0 con 0x7ff6000fdee0 2024-04-16T07:32:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.360+0000 7ff5f6a00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 0x7ff5e40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.361+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7ff5ec08fda0 con 0x7ff6000fdee0 2024-04-16T07:32:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.361+0000 7ff5fea00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 0x7ff5e40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.362+0000 7ff5fea00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 0x7ff5e40710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00041d0 tx=0x7ff5f0009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:36.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.367+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff5ec058aa0 con 0x7ff6000fdee0 2024-04-16T07:32:36.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:36.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:36.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:36.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:36.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:36.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:36.441+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:36.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.716+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff600105860 con 0x7ff6000fdee0 2024-04-16T07:32:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.717+0000 7ff5f6a00700 1 -- 172.21.15.26:0/1544059505 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff5ec05c160 con 0x7ff6000fdee0 2024-04-16T07:32:36.716 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:32:36.718 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 msgr2=0x7ff5e40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 0x7ff5e40710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00041d0 tx=0x7ff5f0009040 comp rx=0 tx=0).stop 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 msgr2=0x7ff600070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00b3b0 tx=0x7ff5ec009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 shutdown_connections 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7ff5e406ec10 0x7ff5e40710c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6000fcc30 0x7ff600071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 --2- 172.21.15.26:0/1544059505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6000fdee0 0x7ff600070310 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.720+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 >> 172.21.15.26:0/1544059505 conn(0x7ff6000f83d0 msgr2=0x7ff6000f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.721+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 shutdown_connections 2024-04-16T07:32:36.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.721+0000 7ff5ffe00700 1 -- 172.21.15.26:0/1544059505 wait complete. 2024-04-16T07:32:36.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:36 smithi026 ceph-mon[92215]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:36.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:36 smithi026 ceph-mon[92215]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:36.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:36 smithi026 ceph-mon[92215]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:36.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:36 smithi104 ceph-mon[81027]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:36.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:36 smithi104 ceph-mon[81027]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:36.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:36 smithi104 ceph-mon[81027]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 -- 172.21.15.26:0/1089744043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e4100070 msgr2=0x7f43e41004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 --2- 172.21.15.26:0/1089744043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e4100070 0x7f43e41004e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f43d0004940 tx=0x7f43d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 -- 172.21.15.26:0/1089744043 shutdown_connections 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 --2- 172.21.15.26:0/1089744043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e4100070 0x7f43e41004e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 --2- 172.21.15.26:0/1089744043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e40fef00 0x7f43e40ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.877+0000 7f43e3200700 1 -- 172.21.15.26:0/1089744043 >> 172.21.15.26:0/1089744043 conn(0x7f43e40fa790 msgr2=0x7f43e40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.878+0000 7f43e3200700 1 -- 172.21.15.26:0/1089744043 shutdown_connections 2024-04-16T07:32:36.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.878+0000 7f43e3200700 1 -- 172.21.15.26:0/1089744043 wait complete. 2024-04-16T07:32:36.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 Processor -- start 2024-04-16T07:32:36.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 -- start start 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4100070 0x7f43e410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41105b0 con 0x7f43e40fef00 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e3200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4110e00 con 0x7f43e4100070 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e1e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e1e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48534/0 (socket says 172.21.15.26:48534) 2024-04-16T07:32:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.879+0000 7f43e1e00700 1 -- 172.21.15.26:0/2025305871 learned_addr learned my addr 172.21.15.26:0/2025305871 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.880+0000 7f43e1400700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4100070 0x7f43e410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.880+0000 7f43e1e00700 1 -- 172.21.15.26:0/2025305871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4100070 msgr2=0x7f43e410ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.880+0000 7f43e1e00700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4100070 0x7f43e410ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.880+0000 7f43e1e00700 1 -- 172.21.15.26:0/2025305871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0005040 con 0x7f43e40fef00 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.880+0000 7f43e1e00700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43d4002a50 tx=0x7f43d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.881+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43d4007cc0 con 0x7f43e40fef00 2024-04-16T07:32:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.881+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e41110e0 con 0x7f43e40fef00 2024-04-16T07:32:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.881+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43d400f040 con 0x7f43e40fef00 2024-04-16T07:32:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.881+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43d4009640 con 0x7f43e40fef00 2024-04-16T07:32:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.881+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e4111620 con 0x7f43e40fef00 2024-04-16T07:32:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.882+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e40fe290 con 0x7f43e40fef00 2024-04-16T07:32:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.884+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f43d40097a0 con 0x7f43e40fef00 2024-04-16T07:32:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.885+0000 7f43da000700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 0x7f43c80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.885+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f43d408e760 con 0x7f43e40fef00 2024-04-16T07:32:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.885+0000 7f43e1400700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 0x7f43c80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.886+0000 7f43e1400700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 0x7f43c80710b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43d0004f90 tx=0x7f43d0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:36.890+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f43d405af10 con 0x7f43e40fef00 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:37.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:37.153+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.180+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f43e41119f0 con 0x7f43e40fef00 2024-04-16T07:32:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.181+0000 7f43da000700 1 -- 172.21.15.26:0/2025305871 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1940 (secure 0 0 0) 0x7f43d405aaa0 con 0x7f43e40fef00 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:e36 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 35 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:56.852323+0000 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:32:37.182 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 85 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=34404} 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:32:37.183 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 34404 members: 34404 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:34404} state up:active seq 22 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:44307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:32:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.184+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 msgr2=0x7f43c80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.184+0000 7f43e3200700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 0x7f43c80710b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43d0004f90 tx=0x7f43d0009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.184+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 msgr2=0x7f43e410fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.184+0000 7f43e3200700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43d4002a50 tx=0x7f43d40046b0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 shutdown_connections 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f43c806ec00 0x7f43c80710b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e40fef00 0x7f43e410fa70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 --2- 172.21.15.26:0/2025305871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4100070 0x7f43e410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 >> 172.21.15.26:0/2025305871 conn(0x7f43e40fa790 msgr2=0x7f43e40fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 shutdown_connections 2024-04-16T07:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.185+0000 7f43e3200700 1 -- 172.21.15.26:0/2025305871 wait complete. 2024-04-16T07:32:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 36 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.338+0000 7f5166400700 1 -- 172.21.15.26:0/1231352564 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 msgr2=0x7f5160100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.338+0000 7f5166400700 1 --2- 172.21.15.26:0/1231352564 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f5160100740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f514c004940 tx=0x7f514c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.338+0000 7f5166400700 1 -- 172.21.15.26:0/1231352564 shutdown_connections 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.338+0000 7f5166400700 1 --2- 172.21.15.26:0/1231352564 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f5160100740 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.339+0000 7f5166400700 1 --2- 172.21.15.26:0/1231352564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 0x7f51600ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.339+0000 7f5166400700 1 -- 172.21.15.26:0/1231352564 >> 172.21.15.26:0/1231352564 conn(0x7f51600fa950 msgr2=0x7f51600fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.339+0000 7f5166400700 1 -- 172.21.15.26:0/1231352564 shutdown_connections 2024-04-16T07:32:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.339+0000 7f5166400700 1 -- 172.21.15.26:0/1231352564 wait complete. 2024-04-16T07:32:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 Processor -- start 2024-04-16T07:32:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 -- start start 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 0x7f516010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51601106b0 con 0x7f51601002d0 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.340+0000 7f5166400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5160110820 con 0x7f51600ff020 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48546/0 (socket says 172.21.15.26:48546) 2024-04-16T07:32:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 -- 172.21.15.26:0/646064135 learned_addr learned my addr 172.21.15.26:0/646064135 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515f400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 0x7f516010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 -- 172.21.15.26:0/646064135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 msgr2=0x7f516010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 0x7f516010fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.341+0000 7f515ea00700 1 -- 172.21.15.26:0/646064135 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c005040 con 0x7f51601002d0 2024-04-16T07:32:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f515ea00700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f514c009150 tx=0x7f514c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514c00e040 con 0x7f51601002d0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f514c00bcc0 con 0x7f51601002d0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5160115220 con 0x7f51601002d0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514c01c630 con 0x7f51601002d0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.342+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51601157e0 con 0x7f51601002d0 2024-04-16T07:32:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.343+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516004c710 con 0x7f51601002d0 2024-04-16T07:32:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.344+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f514c013420 con 0x7f51601002d0 2024-04-16T07:32:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.345+0000 7f5157400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 0x7f51440710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.345+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f514c08f620 con 0x7f51601002d0 2024-04-16T07:32:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.345+0000 7f515f400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 0x7f51440710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.346+0000 7f515f400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 0x7f51440710b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5150007fd0 tx=0x7f5150009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:37.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.350+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f514c05be50 con 0x7f51601002d0 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:37.441+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:37.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.622+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51601045f0 con 0x7f514406ec00 2024-04-16T07:32:37.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.624+0000 7f5157400700 1 -- 172.21.15.26:0/646064135 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f51601045f0 con 0x7f514406ec00 2024-04-16T07:32:37.623 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:32:37.623 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [ 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osd", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mds", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mon" 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Currently upgrading grafana daemons", 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:32:37.624 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 msgr2=0x7f51440710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 0x7f51440710b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5150007fd0 tx=0x7f5150009040 comp rx=0 tx=0).stop 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 msgr2=0x7f51601100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f514c009150 tx=0x7f514c009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 shutdown_connections 2024-04-16T07:32:37.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f514406ec00 0x7f51440710b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51600ff020 0x7f516010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 --2- 172.21.15.26:0/646064135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51601002d0 0x7f51601100b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 >> 172.21.15.26:0/646064135 conn(0x7f51600fa950 msgr2=0x7f51600fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.627+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 shutdown_connections 2024-04-16T07:32:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.628+0000 7f5166400700 1 -- 172.21.15.26:0/646064135 wait complete. 2024-04-16T07:32:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 -- 172.21.15.26:0/3624588237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c0709a0 msgr2=0x7f447c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 --2- 172.21.15.26:0/3624588237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c0709a0 0x7f447c070e10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4468004940 tx=0x7f4468009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 -- 172.21.15.26:0/3624588237 shutdown_connections 2024-04-16T07:32:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 --2- 172.21.15.26:0/3624588237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c0709a0 0x7f447c070e10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 --2- 172.21.15.26:0/3624588237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c071f50 0x7f447c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.795+0000 7f4483200700 1 -- 172.21.15.26:0/3624588237 >> 172.21.15.26:0/3624588237 conn(0x7f447c0f8530 msgr2=0x7f447c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:37.796 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:37 smithi026 ceph-mon[92215]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:37.796 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:37 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1544059505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:37.796 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:37 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2025305871' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:32:37.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.796+0000 7f4483200700 1 -- 172.21.15.26:0/3624588237 shutdown_connections 2024-04-16T07:32:37.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.796+0000 7f4483200700 1 -- 172.21.15.26:0/3624588237 wait complete. 2024-04-16T07:32:37.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.797+0000 7f4483200700 1 Processor -- start 2024-04-16T07:32:37.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.797+0000 7f4483200700 1 -- start start 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.797+0000 7f4483200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 0x7f447c1076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f4483200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f4483200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1092e0 con 0x7f447c071f50 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f4483200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c108120 con 0x7f447c0709a0 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f447b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f447b400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48554/0 (socket says 172.21.15.26:48554) 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f447b400700 1 -- 172.21.15.26:0/3431399448 learned_addr learned my addr 172.21.15.26:0/3431399448 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.798+0000 7f447be00700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 0x7f447c1076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f447b400700 1 -- 172.21.15.26:0/3431399448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 msgr2=0x7f447c1076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:37.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f447b400700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 0x7f447c1076a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f447b400700 1 -- 172.21.15.26:0/3431399448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468005040 con 0x7f447c071f50 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f447be00700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 0x7f447c1076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f447b400700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f446800b3b0 tx=0x7f4468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446800e040 con 0x7f447c071f50 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4468004290 con 0x7f447c071f50 2024-04-16T07:32:37.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c1083a0 con 0x7f447c071f50 2024-04-16T07:32:37.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.799+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446801c630 con 0x7f447c071f50 2024-04-16T07:32:37.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.800+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c122640 con 0x7f447c071f50 2024-04-16T07:32:37.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.801+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c05f5f0 con 0x7f447c071f50 2024-04-16T07:32:37.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.802+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f446800bbf0 con 0x7f447c071f50 2024-04-16T07:32:37.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.803+0000 7f4473e00700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 0x7f44600710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:32:37.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.803+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f446808fe30 con 0x7f447c071f50 2024-04-16T07:32:37.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.803+0000 7f447be00700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 0x7f44600710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:32:37.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.804+0000 7f447be00700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 0x7f44600710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f446c009fd0 tx=0x7f446c009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:32:37.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:37.808+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4468058ab0 con 0x7f447c071f50 2024-04-16T07:32:37.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:37 smithi104 ceph-mon[81027]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:37.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:37 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1544059505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:32:37.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:37 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2025305871' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:38.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:38.153+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.172+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f447c04bed0 con 0x7f447c071f50 2024-04-16T07:32:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.173+0000 7f4473e00700 1 -- 172.21.15.26:0/3431399448 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f446805c170 con 0x7f447c071f50 2024-04-16T07:32:38.172 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.176+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 msgr2=0x7f44600710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.176+0000 7f4483200700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 0x7f44600710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f446c009fd0 tx=0x7f446c009f00 comp rx=0 tx=0).stop 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.176+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 msgr2=0x7f447c107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.176+0000 7f4483200700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f446800b3b0 tx=0x7f4468009e30 comp rx=0 tx=0).stop 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 shutdown_connections 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f446006ec10 0x7f44600710c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f447c0709a0 0x7f447c1076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 --2- 172.21.15.26:0/3431399448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f447c071f50 0x7f447c107be0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:32:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 >> 172.21.15.26:0/3431399448 conn(0x7f447c0f8530 msgr2=0x7f447c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:32:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 shutdown_connections 2024-04-16T07:32:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:32:38.177+0000 7f4483200700 1 -- 172.21.15.26:0/3431399448 wait complete. 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:38.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:38.442+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:38 smithi104 ceph-mon[81027]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:38 smithi104 ceph-mon[81027]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:38.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:38 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3431399448' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:32:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:38 smithi026 ceph-mon[92215]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:39.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:38 smithi026 ceph-mon[92215]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:32:39.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:38 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3431399448' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:32:39.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:39.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:39.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:39.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:39.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:39.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:39.154+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:39.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:39.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:39.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:39.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:39.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:39.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:39.442+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:40.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:40.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:40.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:40.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:40.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:40.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:40.154+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:40.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:40.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:40.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:40.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:40.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:40.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:40.442+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:40.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:40 smithi104 ceph-mon[81027]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:40 smithi026 ceph-mon[92215]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:41.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:41.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:41.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:41.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:41.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:41.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:41.154+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:41.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:41.442+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:42.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:42.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:42.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:42.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:42.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:42.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:42.154+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:42.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:42.443+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:42.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:42 smithi026 ceph-mon[92215]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:42 smithi104 ceph-mon[81027]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:43.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.154+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:43.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.155+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:43.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.155+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:43.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.155+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:43.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.155+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:43.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:43.155+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:43.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:43.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:43.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:43.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:43.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:43.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:43.443+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 39) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:44.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:44.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:44.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:44.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.155+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:44.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.156+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 8 ==== client_session(renewcaps seq 39) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:32:44.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:44.156+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 39) v1 from mds.0 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:44.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:44.443+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:44.783 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:44 smithi026 ceph-mon[92215]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:44.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:44 smithi104 ceph-mon[81027]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:45.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:45.156+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:45.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:45.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:45.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:45.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:45.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:45.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:45.443+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:46.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:46.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:46.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:46.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:46.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:46.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:46.155+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:46.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:46.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:46.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:46.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:46.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:46.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:46.444+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:46.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:46 smithi026 ceph-mon[92215]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:46.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:46 smithi104 ceph-mon[81027]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:47.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:47.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:47.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:47.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:47.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:47.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:47.156+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:47.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:47.444+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:48.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:48.156+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:48.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:48.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:48.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:48.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:48.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:48.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:48.444+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:48.812 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:48 smithi026 ceph-mon[92215]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:48.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:48 smithi104 ceph-mon[81027]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:49.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:49.156+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:49.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:49.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:49.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:49.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:49.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:49.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:49.444+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:50.156+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:50.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:50.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:50.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:50.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:50.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:50.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:50.445+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:50.625 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:50.626 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:50 smithi026 ceph-mon[92215]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:32:50.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:50 smithi104 ceph-mon[81027]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:51.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.156+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:51.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.157+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:51.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.157+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:51.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.157+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:51.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.157+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:51.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:51.157+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:51.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:51.445+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:52.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:52.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:52.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:52.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:52.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:52.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:52.157+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:52.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:52.173 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:52.173 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:51 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:52.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:52.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:52.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:51 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:32:52.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:52.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 39) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.445+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.446+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 8 ==== client_session(renewcaps seq 39) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:32:52.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:52.446+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 39) v1 from mds.0 2024-04-16T07:32:53.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:53.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:53.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:53.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:53.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:53.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:53.157+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:53.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:52 smithi026 ceph-mon[92215]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:53.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:52 smithi104 ceph-mon[81027]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:53.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:53.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:53.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:53.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:53.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:53.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:53.445+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:54.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:54.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:54.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:54.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:54.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:54.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:54.157+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:54.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:54.446+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:55.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:55.157+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:55.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:54 smithi026 ceph-mon[92215]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:54 smithi104 ceph-mon[81027]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:55.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:55.446+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:56.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:56.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:56.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:56.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:56.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:56.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:56.158+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:56.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:56.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:56.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:56.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:56.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:56.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:56.446+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:57.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:56 smithi026 ceph-mon[92215]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:57.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:57.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:57.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:57.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:57.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:57.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:57.158+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:57.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:56 smithi104 ceph-mon[81027]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:57.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:57.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:57.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:57.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:57.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:57.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:57.447+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:58.158+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:58.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:58.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:57 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:58.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:58.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:57 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:58.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:58.447+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:32:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:32:59.158+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:32:59.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:32:58 smithi026 ceph-mon[92215]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:32:58 smithi104 ceph-mon[81027]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:32:59.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:32:59.447+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:00.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.158+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:00.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.158+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:00.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.158+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:00.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.159+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:00.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.159+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:00.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:00.159+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:00.307 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:00.307 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:00 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:00.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:00.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:00 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:00.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:00.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:00.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:00.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:00.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:00.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:00.447+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:01.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:01.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:01.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:01.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:01.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:01.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:01.159+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:01 smithi026 ceph-mon[92215]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:01.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:01.448+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:01.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:01 smithi104 ceph-mon[81027]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:02.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.159+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:02.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.159+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:02.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.159+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:02.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.159+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:02.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.160+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:02.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:02.160+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:02.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:02.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:02.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:02.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:02.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:02.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:02.448+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:03.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:03.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:03.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:03.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 320 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:03.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:03.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.160+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:03.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140822 2024-04-16T07:33:03.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 646, success, outsize: 120 2024-04-16T07:33:03.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.179+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.179+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.179+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.179+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.179+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140910 2024-04-16T07:33:03.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 648, success, outsize: 120 2024-04-16T07:33:03.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.202+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.202+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.202+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.202+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.202+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141001 2024-04-16T07:33:03.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 650, success, outsize: 120 2024-04-16T07:33:03.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.227+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.227+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.227+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.227+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.227+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141098 2024-04-16T07:33:03.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 652, success, outsize: 120 2024-04-16T07:33:03.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.256+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.256+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.256+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.256+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.256+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:03 smithi026 ceph-mon[92215]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:03.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141192 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 654, success, outsize: 120 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.280+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.280+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.280+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.280+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.280+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141280 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 656, success, outsize: 120 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.304+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.304+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.304+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.304+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.304+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141371 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 658, success, outsize: 120 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.330+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.330+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.330+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.330+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.330+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141468 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 660, success, outsize: 120 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.360+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.360+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.360+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.360+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.360+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141562 2024-04-16T07:33:03.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 662, success, outsize: 120 2024-04-16T07:33:03.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.388+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.388+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.388+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.388+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.388+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141650 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 664, success, outsize: 120 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.415+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.415+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.415+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.415+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.415+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141741 2024-04-16T07:33:03.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 666, success, outsize: 120 2024-04-16T07:33:03.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.440+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.440+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.440+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.440+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.440+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:03.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:03.448+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:03.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141838 2024-04-16T07:33:03.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 668, success, outsize: 120 2024-04-16T07:33:03.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.468+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.468+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.468+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.468+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.468+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141932 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 670, success, outsize: 120 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.496+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.496+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.496+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.496+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.496+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142020 2024-04-16T07:33:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 672, success, outsize: 120 2024-04-16T07:33:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.524+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.524+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.524+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.524+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.524+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142111 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 674, success, outsize: 120 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.553+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.553+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.553+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.553+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.553+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142208 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.582+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 676, success, outsize: 120 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.582+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.582+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.583+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.583+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:03 smithi104 ceph-mon[81027]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:03.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142302 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 678, success, outsize: 120 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.609+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.609+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.609+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.609+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.609+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142390 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 680, success, outsize: 120 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.637+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.637+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.637+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.637+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.637+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142481 2024-04-16T07:33:03.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 682, success, outsize: 120 2024-04-16T07:33:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.661+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.661+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.661+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.661+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.661+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142578 2024-04-16T07:33:03.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 684, success, outsize: 120 2024-04-16T07:33:03.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.691+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.691+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.691+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.691+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.691+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142672 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 686, success, outsize: 120 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.715+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.715+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.715+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.715+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.715+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142760 2024-04-16T07:33:03.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 688, success, outsize: 120 2024-04-16T07:33:03.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.741+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.741+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.741+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.741+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.741+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142851 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 690, success, outsize: 120 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.769+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.769+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.769+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.769+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.769+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142948 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 692, success, outsize: 120 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.799+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.799+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.799+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.799+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.799+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143042 2024-04-16T07:33:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 694, success, outsize: 120 2024-04-16T07:33:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.828+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.828+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.828+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.828+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.828+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143130 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 696, success, outsize: 120 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.851+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.851+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.851+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.851+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.851+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143221 2024-04-16T07:33:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 698, success, outsize: 120 2024-04-16T07:33:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.874+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.874+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.874+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.874+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.874+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143318 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr: unique: 700, success, outsize: 120 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.904+0000 7fbf94c4e300 8 client.14524 _ll_getattr 0x1.head 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.904+0000 7fbf94c4e300 10 client.14524 _getattr mask pAsLsXsFs issued=1 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.904+0000 7fbf94c4e300 10 client.14524 fill_stat on 0x1 snap/devhead mode 041777 mtime 2024-04-16T07:19:43.615094+0000 ctime 2024-04-16T07:20:33.192545+0000 2024-04-16T07:33:03.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.904+0000 7fbf94c4e300 3 client.14524 ll_getattr 0x1.head = 0 2024-04-16T07:33:03.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:03.904+0000 7fbf94c4e300 8 client.14524 _ll_forget 0x1 1 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 40) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:33:04.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.160+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.161+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 9 ==== client_session(renewcaps seq 40) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:33:04.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:04.161+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 40) v1 from mds.0 2024-04-16T07:33:04.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:04.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:04.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:04.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:04.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:04.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:04.448+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:05.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:05.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:05.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:05.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:05.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:05.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.161+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:05.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:05.307+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 22 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf8c01d050 con 0x7fbf70008390 2024-04-16T07:33:05.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.307+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 22 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d0010aa0 con 0x55fc3f5009a0 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:05.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:05.448+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:05.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:06.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:06.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:06.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:06.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:06.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:06.161+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:06.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:06.449+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T07:33:06.513 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T07:33:06.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.515 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:06.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T07:33:06.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T07:33:06.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:33:06.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:07.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:07.161+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:07.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:07.449+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:07.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:07 smithi026 ceph-mon[92215]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:33:07.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:07 smithi026 ceph-mon[92215]: Upgrade: Finalizing container_image settings 2024-04-16T07:33:07.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:07 smithi026 ceph-mon[92215]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:07.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:07 smithi026 ceph-mon[92215]: Upgrade: Complete! 2024-04-16T07:33:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:07 smithi104 ceph-mon[81027]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T07:33:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:07 smithi104 ceph-mon[81027]: Upgrade: Finalizing container_image settings 2024-04-16T07:33:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:07 smithi104 ceph-mon[81027]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:07.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:07 smithi104 ceph-mon[81027]: Upgrade: Complete! 2024-04-16T07:33:08.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:08.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:08.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:08.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:08.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:08.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:08.162+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:08.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 -- 172.21.15.26:0/1217469722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05640ffd50 msgr2=0x7f0564102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:08.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 --2- 172.21.15.26:0/1217469722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05640ffd50 0x7f0564102170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f054c004940 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 -- 172.21.15.26:0/1217469722 shutdown_connections 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 --2- 172.21.15.26:0/1217469722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05640ffd50 0x7f0564102170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 --2- 172.21.15.26:0/1217469722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0564066dc0 0x7f05640ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 -- 172.21.15.26:0/1217469722 >> 172.21.15.26:0/1217469722 conn(0x7f05640f72a0 msgr2=0x7f05640f96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 -- 172.21.15.26:0/1217469722 shutdown_connections 2024-04-16T07:33:08.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.352+0000 7f0569800700 1 -- 172.21.15.26:0/1217469722 wait complete. 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.353+0000 7f0569800700 1 Processor -- start 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.353+0000 7f0569800700 1 -- start start 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0569800700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0569800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05640ffd50 0x7f056410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0569800700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05641105a0 con 0x7f0564066dc0 2024-04-16T07:33:08.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0569800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0564110df0 con 0x7f05640ffd50 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0563e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0563e00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59762/0 (socket says 172.21.15.26:59762) 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.354+0000 7f0563e00700 1 -- 172.21.15.26:0/2717147241 learned_addr learned my addr 172.21.15.26:0/2717147241 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f055be00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05640ffd50 0x7f056410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0563e00700 1 -- 172.21.15.26:0/2717147241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05640ffd50 msgr2=0x7f056410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:08.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0563e00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05640ffd50 0x7f056410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0563e00700 1 -- 172.21.15.26:0/2717147241 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c005040 con 0x7f0564066dc0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0563e00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0554002a50 tx=0x7f0554004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0554007cc0 con 0x7f0564066dc0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f055400f040 con 0x7f0564066dc0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f055400da40 con 0x7f0564066dc0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.355+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05641110d0 con 0x7f0564066dc0 2024-04-16T07:33:08.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.356+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05641115c0 con 0x7f0564066dc0 2024-04-16T07:33:08.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.357+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05640faf20 con 0x7f0564066dc0 2024-04-16T07:33:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.358+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f05540097c0 con 0x7f0564066dc0 2024-04-16T07:33:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.359+0000 7f0560c00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 0x7f05480710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.359+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f055408e7c0 con 0x7f0564066dc0 2024-04-16T07:33:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.359+0000 7f055be00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 0x7f05480710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.360+0000 7f055be00700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 0x7f05480710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f054c004f90 tx=0x7f054c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:08.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.365+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f055405af90 con 0x7f0564066dc0 2024-04-16T07:33:08.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:08.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:08.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:08.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:08.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:08.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:08.449+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:08.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:08 smithi026 ceph-mon[92215]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:08.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:08 smithi104 ceph-mon[81027]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.619+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0564111990 con 0x7f054806ec00 2024-04-16T07:33:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.621+0000 7f0560c00700 1 -- 172.21.15.26:0/2717147241 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0564111990 con 0x7f054806ec00 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 msgr2=0x7f05480710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 0x7f05480710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f054c004f90 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 msgr2=0x7f056410fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0554002a50 tx=0x7f0554004600 comp rx=0 tx=0).stop 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 shutdown_connections 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f054806ec00 0x7f05480710b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0564066dc0 0x7f056410fa60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 --2- 172.21.15.26:0/2717147241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05640ffd50 0x7f056410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 >> 172.21.15.26:0/2717147241 conn(0x7f05640f72a0 msgr2=0x7f05640fb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 shutdown_connections 2024-04-16T07:33:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:08.624+0000 7f0569800700 1 -- 172.21.15.26:0/2717147241 wait complete. 2024-04-16T07:33:09.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:09.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:09.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:09.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:09.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:09.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:09.162+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:09 smithi026 ceph-mon[92215]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:09.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:09.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:09.450+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:09.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:09 smithi104 ceph-mon[81027]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:09.959 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:10.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:10.162+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:10.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:10.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:10.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:10.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:10.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:10.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:10.450+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:10 smithi026 ceph-mon[92215]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:11.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:10 smithi104 ceph-mon[81027]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:11.162+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:11.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:11.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:11.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:11.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:11.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:11.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:11.450+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:12.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.162+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:12.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.162+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:12.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.162+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:12.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.162+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:12.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.163+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:12.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:12.163+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:12.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:12.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:33:12.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 40) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.450+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.451+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 9 ==== client_session(renewcaps seq 40) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:33:12.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:12.451+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 40) v1 from mds.0 2024-04-16T07:33:13.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:12 smithi026 ceph-mon[92215]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:12 smithi104 ceph-mon[81027]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:13.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:13.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:13.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:13.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:13.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:13.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:13.163+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:13.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:13.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:13.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:13.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:13.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:13.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:13.451+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:14.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:14.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:14.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:14.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:14.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:14.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:14.164+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:14.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 -- 172.21.15.26:0/1640326498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070c10 msgr2=0x7f0b7c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:14.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 --2- 172.21.15.26:0/1640326498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070c10 0x7f0b7c109170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0b68004970 tx=0x7f0b68009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:14.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 -- 172.21.15.26:0/1640326498 shutdown_connections 2024-04-16T07:33:14.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 --2- 172.21.15.26:0/1640326498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070c10 0x7f0b7c109170 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 --2- 172.21.15.26:0/1640326498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070300 0x7f0b7c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 -- 172.21.15.26:0/1640326498 >> 172.21.15.26:0/1640326498 conn(0x7f0b7c0f8970 msgr2=0x7f0b7c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 -- 172.21.15.26:0/1640326498 shutdown_connections 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.586+0000 7f0b84600700 1 -- 172.21.15.26:0/1640326498 wait complete. 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 Processor -- start 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 -- start start 2024-04-16T07:33:14.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:14.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070c10 0x7f0b7c116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:14.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c1168e0 con 0x7f0b7c070300 2024-04-16T07:33:14.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.587+0000 7f0b84600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c10fcb0 con 0x7f0b7c070c10 2024-04-16T07:33:14.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:14.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59780/0 (socket says 172.21.15.26:59780) 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 -- 172.21.15.26:0/3253716655 learned_addr learned my addr 172.21.15.26:0/3253716655 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81200700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070c10 0x7f0b7c116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 -- 172.21.15.26:0/3253716655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070c10 msgr2=0x7f0b7c116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070c10 0x7f0b7c116290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.588+0000 7f0b81c00700 1 -- 172.21.15.26:0/3253716655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b68005040 con 0x7f0b7c070300 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b81c00700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b70002a50 tx=0x7f0b700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b70007cc0 con 0x7f0b7c070300 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b7000f040 con 0x7f0b7c070300 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b70009640 con 0x7f0b7c070300 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c10ff90 con 0x7f0b7c070300 2024-04-16T07:33:14.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.589+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c1104a0 con 0x7f0b7c070300 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.590+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c071a40 con 0x7f0b7c070300 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.591+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b700097a0 con 0x7f0b7c070300 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.592+0000 7f0b79600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 0x7f0b640710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.593+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f0b7008e760 con 0x7f0b7c070300 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.593+0000 7f0b81200700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 0x7f0b640710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.593+0000 7f0b81200700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 0x7f0b640710c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fd0 tx=0x7f0b68009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:14.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.598+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b7005af10 con 0x7f0b7c070300 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (58s) 16s ago 20m 13.8M - 0.25.0 c8568f914cd2 2e8fc2d1047d 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (7m) 16s ago 20m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (6m) 103s ago 19m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (23s) 16s ago 20m 50.5M - 9.4.12 2bacad6d85d8 46b95e236872 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (3m) 16s ago 13m 21.5M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:33:14.383 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (3m) 16s ago 13m 12.0M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:33:14.384 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (2m) 103s ago 13m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:33:14.384 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (2m) 103s ago 13m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 d300ba6bee79 2024-04-16T07:33:14.384 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (9m) 16s ago 22m 521M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:33:14.384 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (8m) 103s ago 19m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (8m) 16s ago 22m 56.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (8m) 103s ago 19m 39.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (113s) 16s ago 20m 7796k - 1.5.0 0da6a335fe13 2562d9ea91f2 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (107s) 103s ago 19m 6689k - 1.5.0 0da6a335fe13 8fdb5a9c24bb 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (6m) 16s ago 18m 53.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (5m) 16s ago 17m 53.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (5m) 16s ago 16m 44.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (4m) 103s ago 16m 52.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (4m) 103s ago 15m 42.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (3m) 103s ago 15m 43.0M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:33:14.385 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (82s) 16s ago 19m 50.3M - 2.43.0 a07b618ecd1d b4054ff04ef2 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.844+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b7c04bf40 con 0x7f0b6406ec10 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.861+0000 7f0b79600700 1 -- 172.21.15.26:0/3253716655 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0b7c04bf40 con 0x7f0b6406ec10 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 msgr2=0x7f0b640710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 0x7f0b640710c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fd0 tx=0x7f0b68009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 msgr2=0x7f0b7c115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b70002a50 tx=0x7f0b700046b0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 shutdown_connections 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b6406ec10 0x7f0b640710c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b7c070300 0x7f0b7c115d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 --2- 172.21.15.26:0/3253716655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b7c070c10 0x7f0b7c116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:14.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 >> 172.21.15.26:0/3253716655 conn(0x7f0b7c0f8970 msgr2=0x7f0b7c0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:14.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 shutdown_connections 2024-04-16T07:33:14.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:13.864+0000 7f0b84600700 1 -- 172.21.15.26:0/3253716655 wait complete. 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:14.451+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:15.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:14 smithi026 ceph-mon[92215]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:15.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:14 smithi026 ceph-mon[92215]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:15.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:14 smithi104 ceph-mon[81027]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:15.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:14 smithi104 ceph-mon[81027]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:15.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade status' 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:15.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:15.163+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:15.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:15.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:15.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:15.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:15.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:15.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:15.451+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:16.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:16.163+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:16.451+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:17.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:16 smithi104 ceph-mon[81027]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:17.109 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:16 smithi026 ceph-mon[92215]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:17.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:17.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:17.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:17.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:17.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:17.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:17.165+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:17.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:17.452+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:17.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.585+0000 7f0fc5c00700 1 -- 172.21.15.26:0/370393056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 msgr2=0x7f0fc0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:17.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.585+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/370393056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc0070d10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0fac004970 tx=0x7f0fac009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.585+0000 7f0fc5c00700 1 -- 172.21.15.26:0/370393056 shutdown_connections 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.585+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/370393056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc0070d10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.586+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/370393056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc0070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.586+0000 7f0fc5c00700 1 -- 172.21.15.26:0/370393056 >> 172.21.15.26:0/370393056 conn(0x7f0fc00f8970 msgr2=0x7f0fc00fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.586+0000 7f0fc5c00700 1 -- 172.21.15.26:0/370393056 shutdown_connections 2024-04-16T07:33:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.586+0000 7f0fc5c00700 1 -- 172.21.15.26:0/370393056 wait complete. 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.586+0000 7f0fc5c00700 1 Processor -- start 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fc5c00700 1 -- start start 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fc5c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fc5c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc011a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fc5c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc011ad20 con 0x7f0fc00708a0 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fc5c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc01140f0 con 0x7f0fc0071f90 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fbea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.587+0000 7f0fbea00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59802/0 (socket says 172.21.15.26:59802) 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbea00700 1 -- 172.21.15.26:0/1421800960 learned_addr learned my addr 172.21.15.26:0/1421800960 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:33:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbe000700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc011a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbea00700 1 -- 172.21.15.26:0/1421800960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 msgr2=0x7f0fc011a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbea00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc011a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbea00700 1 -- 172.21.15.26:0/1421800960 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac005040 con 0x7f0fc00708a0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbe000700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc011a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.588+0000 7f0fbea00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0002a50 tx=0x7f0fb00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.589+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fb0007cc0 con 0x7f0fc00708a0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.589+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fb000f040 con 0x7f0fc00708a0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.589+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fb0009640 con 0x7f0fc00708a0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.589+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc01143d0 con 0x7f0fc00708a0 2024-04-16T07:33:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.589+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc0114910 con 0x7f0fc00708a0 2024-04-16T07:33:17.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.590+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc0071aa0 con 0x7f0fc00708a0 2024-04-16T07:33:17.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.591+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0fb00097c0 con 0x7f0fc00708a0 2024-04-16T07:33:17.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.592+0000 7f0fb6a00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 0x7f0fa40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:17.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.593+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f0fb008e6b0 con 0x7f0fc00708a0 2024-04-16T07:33:17.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.593+0000 7f0fbe000700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 0x7f0fa40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:17.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.594+0000 7f0fbe000700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 0x7f0fa40710c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00b9c0 tx=0x7f0fac009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:17.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.598+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0fb005af00 con 0x7f0fc00708a0 2024-04-16T07:33:17.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.856+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fc004bf40 con 0x7f0fa406ec10 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.860+0000 7f0fb6a00700 1 -- 172.21.15.26:0/1421800960 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0fc004bf40 con 0x7f0fa406ec10 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": null, 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": false, 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "", 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": null, 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "", 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-04-16T07:33:17.859 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 msgr2=0x7f0fa40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 0x7f0fa40710c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00b9c0 tx=0x7f0fac009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 msgr2=0x7f0fc011a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0002a50 tx=0x7f0fb00046b0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 shutdown_connections 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0fa406ec10 0x7f0fa40710c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fc00708a0 0x7f0fc011a190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 --2- 172.21.15.26:0/1421800960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc0071f90 0x7f0fc011a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.862+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 >> 172.21.15.26:0/1421800960 conn(0x7f0fc00f8970 msgr2=0x7f0fc00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.863+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 shutdown_connections 2024-04-16T07:33:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:17.863+0000 7f0fc5c00700 1 -- 172.21.15.26:0/1421800960 wait complete. 2024-04-16T07:33:18.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:18.164+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:18.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:18.452+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:18.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph health detail' 2024-04-16T07:33:19.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:18 smithi026 ceph-mon[92215]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:19.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:18 smithi026 ceph-mon[92215]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:19.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:18 smithi104 ceph-mon[81027]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:19.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:18 smithi104 ceph-mon[81027]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:33:19.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:19.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:19.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:19.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:19.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:19.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:19.164+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:19.208 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:19.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:19.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:19.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:19.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:19.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:19.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:19.452+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:20.014 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:20.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:20.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:20.164+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:20.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:20.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:20.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:20.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:20.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:20.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:20.453+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:21.052 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:20 smithi026 ceph-mon[92215]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:21.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:20 smithi104 ceph-mon[81027]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:21.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:21.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:21.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:21.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:21.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:21.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:21.165+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:21.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 -- 172.21.15.26:0/1802429917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c1037c0 msgr2=0x7f303c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 --2- 172.21.15.26:0/1802429917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c1037c0 0x7f303c0fce30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3028004940 tx=0x7f3028009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 -- 172.21.15.26:0/1802429917 shutdown_connections 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 --2- 172.21.15.26:0/1802429917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c1037c0 0x7f303c0fce30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 --2- 172.21.15.26:0/1802429917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c102e20 0x7f303c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 -- 172.21.15.26:0/1802429917 >> 172.21.15.26:0/1802429917 conn(0x7f303c0f89b0 msgr2=0x7f303c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 -- 172.21.15.26:0/1802429917 shutdown_connections 2024-04-16T07:33:21.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.363+0000 7f3044400700 1 -- 172.21.15.26:0/1802429917 wait complete. 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.364+0000 7f3044400700 1 Processor -- start 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.364+0000 7f3044400700 1 -- start start 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.364+0000 7f3044400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.364+0000 7f3044400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c1037c0 0x7f303c116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3044400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c116760 con 0x7f303c102e20 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3044400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c10fd80 con 0x7f303c1037c0 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:21.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54014/0 (socket says 172.21.15.26:54014) 2024-04-16T07:33:21.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 -- 172.21.15.26:0/839076384 learned_addr learned my addr 172.21.15.26:0/839076384 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:33:21.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041000700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c1037c0 0x7f303c116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 -- 172.21.15.26:0/839076384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c1037c0 msgr2=0x7f303c116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c1037c0 0x7f303c116110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.365+0000 7f3041a00700 1 -- 172.21.15.26:0/839076384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028005040 con 0x7f303c102e20 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3041a00700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3030002a50 tx=0x7f30300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3030007cc0 con 0x7f303c102e20 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f303000f040 con 0x7f303c102e20 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303000db70 con 0x7f303c102e20 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c110060 con 0x7f303c102e20 2024-04-16T07:33:21.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.366+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c1105a0 con 0x7f303c102e20 2024-04-16T07:33:21.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.368+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c1018b0 con 0x7f303c102e20 2024-04-16T07:33:21.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.369+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f303000dd10 con 0x7f303c102e20 2024-04-16T07:33:21.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.369+0000 7f3039600700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 0x7f30240710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:21.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.370+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f303008dff0 con 0x7f303c102e20 2024-04-16T07:33:21.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.370+0000 7f3041000700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 0x7f30240710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:21.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.371+0000 7f3041000700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 0x7f30240710b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f303c1114b0 tx=0x7f3028009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:21.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.375+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3030056cd0 con 0x7f303c102e20 2024-04-16T07:33:21.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:21.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:21.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:21.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:21.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:21.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:21.453+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:21.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.695+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f303c109f20 con 0x7f303c102e20 2024-04-16T07:33:21.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.696+0000 7f3039600700 1 -- 172.21.15.26:0/839076384 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f303005a390 con 0x7f303c102e20 2024-04-16T07:33:21.696 INFO:teuthology.orchestra.run.smithi026.stdout:HEALTH_OK 2024-04-16T07:33:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.699+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 msgr2=0x7f30240710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.699+0000 7f3044400700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 0x7f30240710b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f303c1114b0 tx=0x7f3028009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.699+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 msgr2=0x7f303c115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.699+0000 7f3044400700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3030002a50 tx=0x7f30300046b0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 shutdown_connections 2024-04-16T07:33:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f302406ec00 0x7f30240710b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f303c102e20 0x7f303c115bd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 --2- 172.21.15.26:0/839076384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f303c1037c0 0x7f303c116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 >> 172.21.15.26:0/839076384 conn(0x7f303c0f89b0 msgr2=0x7f303c102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 shutdown_connections 2024-04-16T07:33:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:21.700+0000 7f3044400700 1 -- 172.21.15.26:0/839076384 wait complete. 2024-04-16T07:33:22.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:21 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/839076384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:33:22.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:21 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/839076384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T07:33:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:22.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:22.165+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:22.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:22.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:22.453+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:22.940 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:23.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:22 smithi104 ceph-mon[81027]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:23.090 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:22 smithi026 ceph-mon[92215]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:23.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:23.165+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:23.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:23.453+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:24.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 41) v5 -- 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.165+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.166+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 10 ==== client_session(renewcaps seq 41) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:33:24.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:24.166+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 41) v1 from mds.0 2024-04-16T07:33:24.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:24.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:24.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:24.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:24.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:24.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:24.454+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 -- 172.21.15.26:0/3634421902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a700656b0 msgr2=0x7f6a70065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 --2- 172.21.15.26:0/3634421902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a700656b0 0x7f6a70065b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c004940 tx=0x7f6a5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 -- 172.21.15.26:0/3634421902 shutdown_connections 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 --2- 172.21.15.26:0/3634421902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a700656b0 0x7f6a70065b20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 --2- 172.21.15.26:0/3634421902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a70102e60 0x7f6a70103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 -- 172.21.15.26:0/3634421902 >> 172.21.15.26:0/3634421902 conn(0x7f6a70072bc0 msgr2=0x7f6a70072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 -- 172.21.15.26:0/3634421902 shutdown_connections 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.879+0000 7f6a74a00700 1 -- 172.21.15.26:0/3634421902 wait complete. 2024-04-16T07:33:24.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.880+0000 7f6a74a00700 1 Processor -- start 2024-04-16T07:33:24.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.880+0000 7f6a74a00700 1 -- start start 2024-04-16T07:33:24.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a74a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a700656b0 0x7f6a70115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:24.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a74a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:24.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a74a00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701168f0 con 0x7f6a70102e60 2024-04-16T07:33:24.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a74a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010fcc0 con 0x7f6a700656b0 2024-04-16T07:33:24.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a6d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:24.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a6d400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54028/0 (socket says 172.21.15.26:54028) 2024-04-16T07:33:24.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a6d400700 1 -- 172.21.15.26:0/2022364665 learned_addr learned my addr 172.21.15.26:0/2022364665 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.881+0000 7f6a6de00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a700656b0 0x7f6a70115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a6d400700 1 -- 172.21.15.26:0/2022364665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a700656b0 msgr2=0x7f6a70115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a6d400700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a700656b0 0x7f6a70115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a6d400700 1 -- 172.21.15.26:0/2022364665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c005040 con 0x7f6a70102e60 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a6d400700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009150 tx=0x7f6a5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c00e040 con 0x7f6a70102e60 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a5c013420 con 0x7f6a70102e60 2024-04-16T07:33:24.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7010ff40 con 0x7f6a70102e60 2024-04-16T07:33:24.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.882+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c01c630 con 0x7f6a70102e60 2024-04-16T07:33:24.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.883+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a70110500 con 0x7f6a70102e60 2024-04-16T07:33:24.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.884+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70107320 con 0x7f6a70102e60 2024-04-16T07:33:24.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.885+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a5c013590 con 0x7f6a70102e60 2024-04-16T07:33:24.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.886+0000 7f6a66000700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 0x7f6a540710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:33:24.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.886+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f6a5c08fce0 con 0x7f6a70102e60 2024-04-16T07:33:24.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.886+0000 7f6a6de00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 0x7f6a540710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:33:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.887+0000 7f6a6de00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 0x7f6a540710c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a60004ec0 tx=0x7f6a60009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:33:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:24.891+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a5c05c490 con 0x7f6a70102e60 2024-04-16T07:33:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:24 smithi026 ceph-mon[92215]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:25.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:24 smithi104 ceph-mon[81027]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:25.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:25.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:25.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:25.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:25.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:25.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:25.166+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.222+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a7005f620 con 0x7f6a70102e60 2024-04-16T07:33:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.223+0000 7f6a66000700 1 -- 172.21.15.26:0/2022364665 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6a5c05c020 con 0x7f6a70102e60 2024-04-16T07:33:25.222 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:33:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T07:33:25.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:33:25.224 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.225+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 msgr2=0x7f6a540710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.225+0000 7f6a74a00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 0x7f6a540710c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a60004ec0 tx=0x7f6a60009040 comp rx=0 tx=0).stop 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.225+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 msgr2=0x7f6a701162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009150 tx=0x7f6a5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 shutdown_connections 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5406ec10 0x7f6a540710c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a700656b0 0x7f6a70115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 --2- 172.21.15.26:0/2022364665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a70102e60 0x7f6a701162a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:33:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 >> 172.21.15.26:0/2022364665 conn(0x7f6a70072bc0 msgr2=0x7f6a700fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:33:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 shutdown_connections 2024-04-16T07:33:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:33:25.226+0000 7f6a74a00700 1 -- 172.21.15.26:0/2022364665 wait complete. 2024-04-16T07:33:25.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.374+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe9a0004750 con 0x55fc3f5009a0 2024-04-16T07:33:25.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.375+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7fe9a0004750 con 0x55fc3f5009a0 2024-04-16T07:33:25.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:25.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:25.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:25.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:25.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:25.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:25.453+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:25.901 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:25 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2022364665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:25.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-16T07:33:26.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:25 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2022364665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:33:26.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.088+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbf5c004750 con 0x7fbf70008390 2024-04-16T07:33:26.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.089+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7fbf5c004750 con 0x7fbf70008390 2024-04-16T07:33:26.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:26.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:26.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:26.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:26.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:26.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:26.166+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:26.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:26.454+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:26.476 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:26.963 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:26 smithi026 ceph-mon[92215]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:27.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:26 smithi104 ceph-mon[81027]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:27.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:27.166+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:27.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:27.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:27.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:27.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:27.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:27.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:27.454+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:28.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:28.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:28.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:28.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:28.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:28.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:28.166+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:28.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:28.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:28.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:28.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:28.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:28.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:28.455+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:28.501 INFO:teuthology.orchestra.run.smithi026.stdout:wait for servicemap items w/ changing names to refresh 2024-04-16T07:33:28.944 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:28 smithi026 ceph-mon[92215]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:29.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:28 smithi104 ceph-mon[81027]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:29.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.166+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:29.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.166+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:29.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.166+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:29.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.167+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:29.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.167+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:29.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:29.167+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:29.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'sleep 60' 2024-04-16T07:33:29.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:29.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:29.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:29.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:29.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:29.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:29.455+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:29.796 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:33:30.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:30.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:30.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:30.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:30.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:30.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:30.167+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:30.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:30.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:30.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:30.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:30.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:30.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:30.455+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:30.961 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:30 smithi026 ceph-mon[92215]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:30 smithi104 ceph-mon[81027]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:31.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:31.167+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:31.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:31.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:31.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:31.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:31.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:31.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:31.456+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:32.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:32.167+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:32.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 41) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:32.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.455+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:32.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.456+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 10 ==== client_session(renewcaps seq 41) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:33:32.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:32.456+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 41) v1 from mds.0 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:33.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:33.168+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:32 smithi026 ceph-mon[92215]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:32 smithi104 ceph-mon[81027]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:33.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:33.456+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:34.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:34.168+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:34.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:34.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:34.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:34.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:34.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:34.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:34.456+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:35.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:35.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:35.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:35.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:35.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:35.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:35.168+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:34 smithi026 ceph-mon[92215]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:34 smithi104 ceph-mon[81027]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:35.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:35.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:35.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:35.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:35.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:35.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:35.457+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:36.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:36.169+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:36.457+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:37.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:37.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:37.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:37.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:37.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:37.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:37.169+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:36 smithi026 ceph-mon[92215]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:36 smithi104 ceph-mon[81027]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:37.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:37.457+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:38.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:38.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:38.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:38.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:38.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:38.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:38.169+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:38.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:38.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:38.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:38.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:38.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:38.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:38.457+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:39.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:39.169+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:38 smithi026 ceph-mon[92215]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:39.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:38 smithi104 ceph-mon[81027]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:39.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:39.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:39.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:39.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:39.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:39.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:39.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.169+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.169+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.169+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.169+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.170+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:40.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:40.170+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:40.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:40.457+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:41.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:41.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:41.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:41.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:41.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:41.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:41.170+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:40 smithi026 ceph-mon[92215]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:41.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:40 smithi104 ceph-mon[81027]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.457+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.457+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:41.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:41.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:42.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:42.170+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:42.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:42.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:42 smithi026 ceph-mon[92215]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:43.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:43.170+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:42 smithi104 ceph-mon[81027]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:43.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:43.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:43.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:43.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:43.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:43.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:43.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:44.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 42) v5 -- 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:44.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.171+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:44.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.172+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 11 ==== client_session(renewcaps seq 42) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:33:44.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:44.172+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 42) v1 from mds.0 2024-04-16T07:33:44.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:44.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:44.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:44.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:44.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:44.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:44.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:45.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:45.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:45.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:45.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:45.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:45.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:45.171+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:45.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:44 smithi026 ceph-mon[92215]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:45.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:44 smithi104 ceph-mon[81027]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:45.458+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:46.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:46.171+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:46.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:46.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:46.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:46.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:46.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:46.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:46.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:47.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.171+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:47.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.171+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:47.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.171+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:47.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.171+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:47.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.172+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:47.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:47.172+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:47.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:46 smithi026 ceph-mon[92215]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:47.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:46 smithi104 ceph-mon[81027]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:47.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:47.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:47.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:47.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:47.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:47.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:47.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:48.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:48.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:48.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:48.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:48.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:48.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:48.172+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:48.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:49.136 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:48 smithi104 ceph-mon[81027]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:49.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:49.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:49.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:49.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:49.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:49.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:49.172+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:48 smithi026 ceph-mon[92215]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:49.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:49.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:50.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:50.173+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:50.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:50.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:50.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:51.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:51.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:51.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:51.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:51.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:51.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:51.173+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:50 smithi026 ceph-mon[92215]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:51.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:50 smithi104 ceph-mon[81027]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:51.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:51.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:51.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:51.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:51.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:51.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:51.460+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:52.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:52.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:52.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:52.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:52.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:52.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:52.173+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 42) v5 -- 0x7fe9ac0066a0 con 0x7fe9b4014240 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:52.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.459+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:52.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.460+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 11 ==== client_session(renewcaps seq 42) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:33:52.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:52.460+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 42) v1 from mds.0 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:53.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:53.173+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:53.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:52 smithi026 ceph-mon[92215]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:53.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:52 smithi104 ceph-mon[81027]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:53.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:53.460+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:54.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:54.173+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:54.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:54.461+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:55.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:55.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:55.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:55.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:55.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:55.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:55.174+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:54 smithi026 ceph-mon[92215]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:55.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:54 smithi104 ceph-mon[81027]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:55.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:55.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:55.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:55.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:55.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:55.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:55.461+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:56.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:56.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:56.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:56.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:56.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:56.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:56.174+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:56.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:56.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:56.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:56.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:56.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:56.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:56.461+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:57.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:57.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:57.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:57.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:57.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:57.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:57.174+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:56 smithi026 ceph-mon[92215]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:57.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:56 smithi104 ceph-mon[81027]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:57.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:57.462+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:58.174+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:58.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:58.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:58.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:58.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:58.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:58.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:58.462+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:59.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:33:58 smithi026 ceph-mon[92215]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:59.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:33:59.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:33:59.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:33:59.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:33:59.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:33:59.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:33:59.175+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:33:59.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:33:58 smithi104 ceph-mon[81027]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:33:59.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:33:59.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:33:59.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:33:59.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:33:59.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:33:59.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:33:59.462+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:00.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:00.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:00.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:00.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:00.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:00.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:00.175+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:00.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:00.462+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:01.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:01.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:01.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:01.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:01.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:01.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:01.175+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:00 smithi026 ceph-mon[92215]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:01.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:00 smithi104 ceph-mon[81027]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:01.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:01.463+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:02.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:02.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:02.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:02.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:02.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:02.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:02.176+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:02.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:02.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:02.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:02.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:02.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:02.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:02.463+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:03.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:03.176+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:02 smithi026 ceph-mon[92215]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:03.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:02 smithi104 ceph-mon[81027]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:03.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:03.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:03.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:03.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:03.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:03.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:03.463+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 43) v5 -- 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:04.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:04.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 12 ==== client_session(renewcaps seq 43) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047f0 con 0x7fbf7000cfc0 2024-04-16T07:34:04.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:04.177+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 43) v1 from mds.0 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:04.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:04.463+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:05.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:05.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:05.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:05.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:05.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:05.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:05.176+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:04 smithi026 ceph-mon[92215]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:04 smithi104 ceph-mon[81027]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:05.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:05.464+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:06.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:06.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:06.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:06.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:06.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:06.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:06.177+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:06.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:06.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:06.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:06.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:06.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:06.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:06.464+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:07.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:07.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:07.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:07.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:07.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:07.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:07.177+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:07.260 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:06 smithi026 ceph-mon[92215]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:06 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:34:07.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:06 smithi104 ceph-mon[81027]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:07.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:06 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:34:07.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:07.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:07.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:07.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:07.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:07.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:07.464+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:08.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:08.177+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:08.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:08.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:08.464+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:08.510 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:08.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:09.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:09.178+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:09 smithi026 ceph-mon[92215]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:09 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:34:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:09 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:34:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:09 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:09.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:09 smithi104 ceph-mon[81027]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:09.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:34:09.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:34:09.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:09 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:34:09.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:09.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:09.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:09.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:09.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:09.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:09.465+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:10.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:10.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:10.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:10.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:10.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:10.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:10.178+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:10.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:10.465+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:11.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:11.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:11.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:11.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:11.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:11.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:11.179+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:11.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:11.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:11.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:11.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:11.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:11.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:11.465+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:11.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:11 smithi026 ceph-mon[92215]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:11.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:11 smithi104 ceph-mon[81027]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:12.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:12.178+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 43) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:12.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:12.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006930 con 0x7fe9b4014240 2024-04-16T07:34:12.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:12.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:12.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 12 ==== client_session(renewcaps seq 43) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:34:12.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:12.466+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 43) v1 from mds.0 2024-04-16T07:34:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:13.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:13.178+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:13.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:13.466+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:13.516 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:13 smithi026 ceph-mon[92215]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:13.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:13 smithi104 ceph-mon[81027]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:14.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:14.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:14.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:14.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:14.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:14.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:14.179+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:14.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:14.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:14.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:14.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:14.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:14.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:14.466+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:15.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:15.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:15.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:15.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:15.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:15.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:15.179+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:15.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:15.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:15.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:15.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:15.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:15.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:15.466+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:15.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:15 smithi026 ceph-mon[92215]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:15.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:15 smithi104 ceph-mon[81027]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:16.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:16.179+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:16.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.466+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:16.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:16.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.466+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:16.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.466+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:16.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.467+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:16.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:16.467+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:17.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:17.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:17.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:17.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:17.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:17.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:17.179+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:17.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:17.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:17.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:17.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:17.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:17.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:17.467+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:17 smithi026 ceph-mon[92215]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:17.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:17 smithi104 ceph-mon[81027]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:18.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:18.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:18.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:18.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:18.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:18.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:18.180+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:18.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:18.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:18.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:18.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:18.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:18.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:18.467+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:19.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:19.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:19.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:19.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:19.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:19.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:19.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:19 smithi104 ceph-mon[81027]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:19.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:19.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:19.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:19.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:19.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:19.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:19.467+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:19 smithi026 ceph-mon[92215]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:20.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:20.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:20.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:20.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:20.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:20.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:20.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:20.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:20.468+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:20.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:21.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:21.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:21.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:21.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:21.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:21.468+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:21 smithi026 ceph-mon[92215]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:21.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:21 smithi104 ceph-mon[81027]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:22.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:22.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:22.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:22.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:22.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:22.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:22.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:22.429 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:22 smithi026 ceph-mon[92215]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:22.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:22.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:22.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:22.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:22.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:22.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:22.468+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:22.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:22 smithi104 ceph-mon[81027]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:23.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:23.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:23.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:23.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:23.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:23.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:23.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:23.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:23.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:23.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:23.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:23.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:23.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:23.468+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 44) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:24.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.181+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:24.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.182+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 13 ==== client_session(renewcaps seq 44) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:34:24.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:24.182+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 44) v1 from mds.0 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:24.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:24.468+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:24.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:24 smithi104 ceph-mon[81027]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:24 smithi026 ceph-mon[92215]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:25.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:25.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:25.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:25.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:25.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:25.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:25.182+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:25.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:25.469+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:26.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:26.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:26.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:26.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:26.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:26.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:26.182+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:26.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:26.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:26.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:26.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:26.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:26.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:26.469+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:26.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:26 smithi104 ceph-mon[81027]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:27.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:26 smithi026 ceph-mon[92215]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:27.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:27.182+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:27.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:27.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:27.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:27.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:27.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:27.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:27.469+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:28.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.182+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:28.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.182+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:28.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.182+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:28.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.182+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:28.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.183+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:28.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:28.183+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:28.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:28.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:28.469+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:28.813 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:28 smithi026 ceph-mon[92215]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:28.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:28 smithi104 ceph-mon[81027]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:29.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:29.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:29.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:29.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:29.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:29.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:29.183+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:29.470+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:30.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:30.183+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:30.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:30.470+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:30.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:30 smithi104 ceph-mon[81027]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:31.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:30 smithi026 ceph-mon[92215]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:31.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:31.183+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:31.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:31.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:31.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:31.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:31.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:31.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:31.470+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:32.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:32.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:32.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:32.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:32.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:32.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:32.184+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 44) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006930 con 0x7fe9b4014240 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:32.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.470+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:32.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.471+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 13 ==== client_session(renewcaps seq 44) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:34:32.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:32.471+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 44) v1 from mds.0 2024-04-16T07:34:32.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T07:34:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:32 smithi026 ceph-mon[92215]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:32 smithi104 ceph-mon[81027]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:33.005 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:34:33.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:33.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:33.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:33.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:33.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:33.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:33.184+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006930 con 0x7fe9b4014240 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:33.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:33.471+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:34.184+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:34.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:34.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:34.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:34.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006930 con 0x7fe9b4014240 2024-04-16T07:34:34.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:34.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:34.471+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:34 smithi026 ceph-mon[92215]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:34.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:34 smithi104 ceph-mon[81027]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:34.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 -- 172.21.15.26:0/823453093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83541035f0 msgr2=0x7f83540fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 --2- 172.21.15.26:0/823453093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83541035f0 0x7f83540fcc60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8340004940 tx=0x7f8340009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 -- 172.21.15.26:0/823453093 shutdown_connections 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 --2- 172.21.15.26:0/823453093 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83541035f0 0x7f83540fcc60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 --2- 172.21.15.26:0/823453093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8354102c50 0x7f8354103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 -- 172.21.15.26:0/823453093 >> 172.21.15.26:0/823453093 conn(0x7f83540f87c0 msgr2=0x7f83540fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 -- 172.21.15.26:0/823453093 shutdown_connections 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.946+0000 7f8359200700 1 -- 172.21.15.26:0/823453093 wait complete. 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.947+0000 7f8359200700 1 Processor -- start 2024-04-16T07:34:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.947+0000 7f8359200700 1 -- start start 2024-04-16T07:34:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8359200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8359200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83541035f0 0x7f8354113ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8359200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8354114540 con 0x7f8354102c50 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8359200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835410d930 con 0x7f83541035f0 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52164/0 (socket says 172.21.15.26:52164) 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 -- 172.21.15.26:0/2442872214 learned_addr learned my addr 172.21.15.26:0/2442872214 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8352a00700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83541035f0 0x7f8354113ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 -- 172.21.15.26:0/2442872214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83541035f0 msgr2=0x7f8354113ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83541035f0 0x7f8354113ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.948+0000 7f8353400700 1 -- 172.21.15.26:0/2442872214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340005040 con 0x7f8354102c50 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f8353400700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f834400c8c0 tx=0x7f83440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834400a980 con 0x7f8354102c50 2024-04-16T07:34:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f834400ce60 con 0x7f8354102c50 2024-04-16T07:34:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835410dc10 con 0x7f8354102c50 2024-04-16T07:34:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83440115e0 con 0x7f8354102c50 2024-04-16T07:34:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.949+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835410e1d0 con 0x7f8354102c50 2024-04-16T07:34:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.950+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83540fe3a0 con 0x7f8354102c50 2024-04-16T07:34:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.951+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8344012400 con 0x7f8354102c50 2024-04-16T07:34:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.953+0000 7f834b400700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 0x7f83380710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.953+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f834408eb30 con 0x7f8354102c50 2024-04-16T07:34:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.953+0000 7f8352a00700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 0x7f83380710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.954+0000 7f8352a00700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 0x7f83380710b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f835410eaf0 tx=0x7f8340009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:34.958+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f834405b2e0 con 0x7f8354102c50 2024-04-16T07:34:35.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:35.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:35.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:35.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:35.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:35.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:35.184+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:35.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.216+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 --> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8354109d20 con 0x7f833806ec00 2024-04-16T07:34:35.229 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (2m) 98s ago 21m 13.8M - 0.25.0 c8568f914cd2 2e8fc2d1047d 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (8m) 98s ago 22m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 5d9ff7c974fd 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi104 smithi104 running (8m) 3m ago 20m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 38c594928017 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (104s) 98s ago 21m 50.5M - 9.4.12 2bacad6d85d8 46b95e236872 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kqpyuh smithi026 running (4m) 98s ago 14m 21.5M - 19.0.0-2884-g218a6914 a4e4811815e4 c0aa860dcd4a 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.kwjewe smithi026 running (4m) 98s ago 14m 12.0M - 19.0.0-2884-g218a6914 a4e4811815e4 85b686fff0b4 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.dpfdje smithi104 running (3m) 3m ago 14m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 c80089ecb780 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi104.zwhiwr smithi104 running (3m) 3m ago 14m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 d300ba6bee79 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.tawexg smithi026 *:8443,9283,8765 running (11m) 98s ago 23m 521M - 19.0.0-2884-g218a6914 a4e4811815e4 3db881c285e0 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi104.uqfljg smithi104 *:8443,9283,8765 running (10m) 3m ago 20m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 1822b776e3fb 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (9m) 98s ago 23m 56.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 86cbf092af9f 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi104 smithi104 running (9m) 3m ago 20m 39.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 08a1ef0af55e 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (3m) 98s ago 22m 7796k - 1.5.0 0da6a335fe13 2562d9ea91f2 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi104 smithi104 *:9100 running (3m) 3m ago 20m 6689k - 1.5.0 0da6a335fe13 8fdb5a9c24bb 2024-04-16T07:34:35.230 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (7m) 98s ago 19m 53.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8c50566034f1 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (7m) 98s ago 18m 53.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 1988d8980b6f 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (6m) 98s ago 18m 44.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 8f70feb33b93 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi104 running (6m) 3m ago 17m 52.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 d9e22e0fb3cc 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi104 running (5m) 3m ago 17m 42.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 37e9774e15c5 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi104 running (5m) 3m ago 16m 43.0M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 ffa3ac0a534d 2024-04-16T07:34:35.231 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (2m) 98s ago 21m 50.3M - 2.43.0 a07b618ecd1d b4054ff04ef2 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.229+0000 7f834b400700 1 -- 172.21.15.26:0/2442872214 <== mgr.34100 v2:172.21.15.26:6800/2662957189 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f8354109d20 con 0x7f833806ec00 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 msgr2=0x7f83380710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 0x7f83380710b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f835410eaf0 tx=0x7f8340009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 msgr2=0x7f83541139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f834400c8c0 tx=0x7f83440044c0 comp rx=0 tx=0).stop 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 shutdown_connections 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f833806ec00 0x7f83380710b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8354102c50 0x7f83541139b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 --2- 172.21.15.26:0/2442872214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83541035f0 0x7f8354113ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 >> 172.21.15.26:0/2442872214 conn(0x7f83540f87c0 msgr2=0x7f83540f97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 shutdown_connections 2024-04-16T07:34:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:35.232+0000 7f8359200700 1 -- 172.21.15.26:0/2442872214 wait complete. 2024-04-16T07:34:35.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:35.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:35.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:35.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006930 con 0x7fe9b4014240 2024-04-16T07:34:35.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:35.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:35.471+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:35.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T07:34:36.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.184+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:36.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.185+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:36.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.185+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:36.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.185+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:36.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.185+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:36.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:36.185+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:36.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:36.472+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:36.512 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:34:36.822 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:36 smithi026 ceph-mon[92215]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:34:36.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:36 smithi026 ceph-mon[92215]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:36.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:36 smithi104 ceph-mon[81027]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T07:34:36.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:36 smithi104 ceph-mon[81027]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:37.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:37.185+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:37.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:37.472+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:38.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:38.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:38.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:38.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:38.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:38.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:38.185+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:38.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:38.472+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:38.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:38 smithi026 ceph-mon[92215]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 -- 172.21.15.26:0/1070170142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d4105030 msgr2=0x7f98d4105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 --2- 172.21.15.26:0/1070170142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d4105030 0x7f98d4105400 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f98c8004940 tx=0x7f98c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 -- 172.21.15.26:0/1070170142 shutdown_connections 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 --2- 172.21.15.26:0/1070170142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d41059d0 0x7f98d40ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 --2- 172.21.15.26:0/1070170142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d4105030 0x7f98d4105400 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 -- 172.21.15.26:0/1070170142 >> 172.21.15.26:0/1070170142 conn(0x7f98d40fabd0 msgr2=0x7f98d40fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 -- 172.21.15.26:0/1070170142 shutdown_connections 2024-04-16T07:34:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.827+0000 7f98dbe00700 1 -- 172.21.15.26:0/1070170142 wait complete. 2024-04-16T07:34:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.828+0000 7f98dbe00700 1 Processor -- start 2024-04-16T07:34:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.828+0000 7f98dbe00700 1 -- start start 2024-04-16T07:34:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98dbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d4105030 0x7f98d4115b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98dbe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98dbe00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d4116720 con 0x7f98d41059d0 2024-04-16T07:34:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98dbe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d410fd00 con 0x7f98d4105030 2024-04-16T07:34:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98d8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98d8a00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45158/0 (socket says 172.21.15.26:45158) 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.829+0000 7f98d8a00700 1 -- 172.21.15.26:0/796587479 learned_addr learned my addr 172.21.15.26:0/796587479 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.830+0000 7f98d9400700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d4105030 0x7f98d4115b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.830+0000 7f98d8a00700 1 -- 172.21.15.26:0/796587479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d4105030 msgr2=0x7f98d4115b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.830+0000 7f98d8a00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d4105030 0x7f98d4115b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.830+0000 7f98d8a00700 1 -- 172.21.15.26:0/796587479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8005040 con 0x7f98d41059d0 2024-04-16T07:34:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.830+0000 7f98d8a00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f98c000eb40 tx=0x7f98c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.831+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c0007cc0 con 0x7f98d41059d0 2024-04-16T07:34:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.831+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98c000f040 con 0x7f98d41059d0 2024-04-16T07:34:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.831+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c000c9d0 con 0x7f98d41059d0 2024-04-16T07:34:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.831+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d410ff80 con 0x7f98d41059d0 2024-04-16T07:34:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.831+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d41104c0 con 0x7f98d41059d0 2024-04-16T07:34:38.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.832+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d4100780 con 0x7f98d41059d0 2024-04-16T07:34:38.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:38 smithi104 ceph-mon[81027]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:38.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.833+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f98c000cb30 con 0x7f98d41059d0 2024-04-16T07:34:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.835+0000 7f98d1600700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 0x7f98bc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.835+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f98c008e2d0 con 0x7f98d41059d0 2024-04-16T07:34:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.835+0000 7f98d9400700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 0x7f98bc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:38.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.836+0000 7f98d9400700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 0x7f98bc0710b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009000 tx=0x7f98c8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:38.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:38.839+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f98c005abd0 con 0x7f98d41059d0 2024-04-16T07:34:39.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.164+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f98d410c130 con 0x7f98d41059d0 2024-04-16T07:34:39.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.165+0000 7f98d1600700 1 -- 172.21.15.26:0/796587479 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f98c005a760 con 0x7f98d41059d0 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:34:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-16T07:34:39.167 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 msgr2=0x7f98bc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 0x7f98bc0710b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009000 tx=0x7f98c8009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 msgr2=0x7f98d41160d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f98c000eb40 tx=0x7f98c000b040 comp rx=0 tx=0).stop 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 shutdown_connections 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f98bc06ec00 0x7f98bc0710b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d4105030 0x7f98d4115b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 --2- 172.21.15.26:0/796587479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98d41059d0 0x7f98d41160d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:39.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 >> 172.21.15.26:0/796587479 conn(0x7f98d40fabd0 msgr2=0x7f98d40fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:39.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.170+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 shutdown_connections 2024-04-16T07:34:39.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:39.171+0000 7f98dbe00700 1 -- 172.21.15.26:0/796587479 wait complete. 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:39.185+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:39.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:39.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:39.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:39.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:39.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:39.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:39.472+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:39.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:39 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/796587479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:39.980 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:39 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/796587479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:40.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-16T07:34:40.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.185+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:40.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.186+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:40.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.186+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:40.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.186+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:40.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.186+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:40.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:40.186+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:40.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:40.473+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:40.560 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:34:40.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:40 smithi104 ceph-mon[81027]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:40.892 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:40 smithi026 ceph-mon[92215]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:41.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:41.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:41.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:41.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:41.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:41.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:41.186+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:41.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:41.473+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:42.186+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:42.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:42.473+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.814+0000 7f4a78400700 1 -- 172.21.15.26:0/1198388302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 msgr2=0x7f4a700ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.814+0000 7f4a78400700 1 --2- 172.21.15.26:0/1198388302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a700ff020 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c004940 tx=0x7f4a5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.814+0000 7f4a78400700 1 -- 172.21.15.26:0/1198388302 shutdown_connections 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 --2- 172.21.15.26:0/1198388302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a700ff020 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 --2- 172.21.15.26:0/1198388302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 0x7f4a701053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:42.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 -- 172.21.15.26:0/1198388302 >> 172.21.15.26:0/1198388302 conn(0x7f4a700faba0 msgr2=0x7f4a700fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 -- 172.21.15.26:0/1198388302 shutdown_connections 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 -- 172.21.15.26:0/1198388302 wait complete. 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.815+0000 7f4a78400700 1 Processor -- start 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a78400700 1 -- start start 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a78400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 0x7f4a70115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a78400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a78400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70116970 con 0x7f4a701059b0 2024-04-16T07:34:42.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a78400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7010fd40 con 0x7f4a70105010 2024-04-16T07:34:42.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a75000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a75000700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45172/0 (socket says 172.21.15.26:45172) 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.816+0000 7f4a75000700 1 -- 172.21.15.26:0/3250939892 learned_addr learned my addr 172.21.15.26:0/3250939892 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a75a00700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 0x7f4a70115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a75000700 1 -- 172.21.15.26:0/3250939892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 msgr2=0x7f4a70115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a75000700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 0x7f4a70115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a75000700 1 -- 172.21.15.26:0/3250939892 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c005040 con 0x7f4a701059b0 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a75000700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009100 tx=0x7f4a5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5c00e040 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5c013420 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.817+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7010ffc0 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.818+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5c01c630 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.818+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a70110580 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.819+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a70100760 con 0x7f4a701059b0 2024-04-16T07:34:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.820+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a5c013590 con 0x7f4a701059b0 2024-04-16T07:34:42.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.821+0000 7f4a6d600700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 0x7f4a580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:42.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.821+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f4a5c08fba0 con 0x7f4a701059b0 2024-04-16T07:34:42.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.821+0000 7f4a75a00700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 0x7f4a580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:42.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.822+0000 7f4a75a00700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 0x7f4a580710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a64004990 tx=0x7f4a64009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:42.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:42.826+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a5c05c3d0 con 0x7f4a701059b0 2024-04-16T07:34:42.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:42 smithi104 ceph-mon[81027]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:42 smithi026 ceph-mon[92215]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:43.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.157+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4a7004bed0 con 0x7f4a701059b0 2024-04-16T07:34:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.158+0000 7f4a6d600700 1 -- 172.21.15.26:0/3250939892 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4a5c058030 con 0x7f4a701059b0 2024-04-16T07:34:43.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.161+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 msgr2=0x7f4a580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:43.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.161+0000 7f4a78400700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 0x7f4a580710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a64004990 tx=0x7f4a64009ee0 comp rx=0 tx=0).stop 2024-04-16T07:34:43.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.161+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 msgr2=0x7f4a70116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.161+0000 7f4a78400700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009100 tx=0x7f4a5c009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 shutdown_connections 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a5806ec10 0x7f4a580710c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70105010 0x7f4a70115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 --2- 172.21.15.26:0/3250939892 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a701059b0 0x7f4a70116320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 >> 172.21.15.26:0/3250939892 conn(0x7f4a700faba0 msgr2=0x7f4a700fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 shutdown_connections 2024-04-16T07:34:43.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:43.162+0000 7f4a78400700 1 -- 172.21.15.26:0/3250939892 wait complete. 2024-04-16T07:34:43.178 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:43.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:43.186+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:43.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:43.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:43.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:43.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:43.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:43.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:43.473+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:43.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:43 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3250939892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:43.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:43 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3250939892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:43.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-16T07:34:44.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.186+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:44.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 45) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 14 ==== client_session(renewcaps seq 45) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:34:44.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:44.187+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 45) v1 from mds.0 2024-04-16T07:34:44.363 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:34:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:44.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:44.474+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:44.725 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:44 smithi026 ceph-mon[92215]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:45.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:44 smithi104 ceph-mon[81027]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:45.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:45.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:45.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:45.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:45.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:45.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:45.187+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:45.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:45.474+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:46.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:46.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:46.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:46.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:46.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:46.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:46.187+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:46.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:46.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:46.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:46.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:46.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:46.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:46.474+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:46.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 -- 172.21.15.26:0/2257999918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c400656b0 msgr2=0x7f6c40065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 --2- 172.21.15.26:0/2257999918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c400656b0 0x7f6c40065b20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6c28004940 tx=0x7f6c28009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 -- 172.21.15.26:0/2257999918 shutdown_connections 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 --2- 172.21.15.26:0/2257999918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c400656b0 0x7f6c40065b20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 --2- 172.21.15.26:0/2257999918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c40102e60 0x7f6c40103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.494+0000 7f6c47200700 1 -- 172.21.15.26:0/2257999918 >> 172.21.15.26:0/2257999918 conn(0x7f6c40072bc0 msgr2=0x7f6c40072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.495+0000 7f6c47200700 1 -- 172.21.15.26:0/2257999918 shutdown_connections 2024-04-16T07:34:46.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.495+0000 7f6c47200700 1 -- 172.21.15.26:0/2257999918 wait complete. 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.495+0000 7f6c47200700 1 Processor -- start 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c47200700 1 -- start start 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c47200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400656b0 0x7f6c40115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c47200700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c47200700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c40116890 con 0x7f6c40102e60 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c47200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4010fc60 con 0x7f6c400656b0 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c37400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c37400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45196/0 (socket says 172.21.15.26:45196) 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.496+0000 7f6c37400700 1 -- 172.21.15.26:0/2843762618 learned_addr learned my addr 172.21.15.26:0/2843762618 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c3fe00700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400656b0 0x7f6c40115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c37400700 1 -- 172.21.15.26:0/2843762618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400656b0 msgr2=0x7f6c40115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c37400700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400656b0 0x7f6c40115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c37400700 1 -- 172.21.15.26:0/2843762618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c28005040 con 0x7f6c40102e60 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c37400700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009150 tx=0x7f6c28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.497+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2800e040 con 0x7f6c40102e60 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.498+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c28013420 con 0x7f6c40102e60 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.498+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4010fee0 con 0x7f6c40102e60 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.498+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2801c630 con 0x7f6c40102e60 2024-04-16T07:34:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.498+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c401104a0 con 0x7f6c40102e60 2024-04-16T07:34:46.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.499+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c401069d0 con 0x7f6c40102e60 2024-04-16T07:34:46.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.500+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c28013590 con 0x7f6c40102e60 2024-04-16T07:34:46.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.501+0000 7f6c3cc00700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 0x7f6c240710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:46.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.502+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f6c2808fc00 con 0x7f6c40102e60 2024-04-16T07:34:46.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.502+0000 7f6c3fe00700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 0x7f6c240710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:46.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.502+0000 7f6c3fe00700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 0x7f6c240710c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007fd0 tx=0x7f6c30009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:46.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.507+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6c2805c430 con 0x7f6c40102e60 2024-04-16T07:34:46.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.838+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6c400fe570 con 0x7f6c40102e60 2024-04-16T07:34:46.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.839+0000 7f6c3cc00700 1 -- 172.21.15.26:0/2843762618 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6c2805bfc0 con 0x7f6c40102e60 2024-04-16T07:34:46.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 msgr2=0x7f6c240710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:46.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 0x7f6c240710c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007fd0 tx=0x7f6c30009040 comp rx=0 tx=0).stop 2024-04-16T07:34:46.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 msgr2=0x7f6c40116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009150 tx=0x7f6c28009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 shutdown_connections 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6c2406ec10 0x7f6c240710c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400656b0 0x7f6c40115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 --2- 172.21.15.26:0/2843762618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6c40102e60 0x7f6c40116240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 >> 172.21.15.26:0/2843762618 conn(0x7f6c40072bc0 msgr2=0x7f6c400facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 shutdown_connections 2024-04-16T07:34:46.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:46.843+0000 7f6c47200700 1 -- 172.21.15.26:0/2843762618 wait complete. 2024-04-16T07:34:46.859 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)" 2024-04-16T07:34:46.859 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:46 smithi026 ceph-mon[92215]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:47.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:46 smithi104 ceph-mon[81027]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:47.187+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:47.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:47.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:47.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:47.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:47.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:47.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:47.474+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:47.594 DEBUG:teuthology.parallel:result is None 2024-04-16T07:34:47.594 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T07:34:47.607 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:34:47.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T07:34:48.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:47 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2843762618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:48.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:47 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2843762618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T07:34:48.136 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/mon.smithi026/config 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:48.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:48.188+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:48.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:48.475+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:48 smithi026 ceph-mon[92215]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:49.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:48 smithi104 ceph-mon[81027]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:49.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:49.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:49.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:49.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:49.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:49.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:49.188+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:49.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:49.475+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:49.939 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:49.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:50.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:50.188+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:50.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 -- 172.21.15.26:0/1443685459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec102e60 msgr2=0x7f97ec103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 --2- 172.21.15.26:0/1443685459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec102e60 0x7f97ec103230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97dc004940 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 -- 172.21.15.26:0/1443685459 shutdown_connections 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 --2- 172.21.15.26:0/1443685459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec0656b0 0x7f97ec065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 --2- 172.21.15.26:0/1443685459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec102e60 0x7f97ec103230 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 -- 172.21.15.26:0/1443685459 >> 172.21.15.26:0/1443685459 conn(0x7f97ec072bc0 msgr2=0x7f97ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 -- 172.21.15.26:0/1443685459 shutdown_connections 2024-04-16T07:34:50.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.297+0000 7f97f2c00700 1 -- 172.21.15.26:0/1443685459 wait complete. 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.298+0000 7f97f2c00700 1 Processor -- start 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.298+0000 7f97f2c00700 1 -- start start 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.298+0000 7f97f2c00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97f2c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec114260 0x7f97ec10dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97f2c00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1147e0 con 0x7f97ec0656b0 2024-04-16T07:34:50.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97f2c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec114950 con 0x7f97ec114260 2024-04-16T07:34:50.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:50.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49274/0 (socket says 172.21.15.26:49274) 2024-04-16T07:34:50.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 -- 172.21.15.26:0/2722652023 learned_addr learned my addr 172.21.15.26:0/2722652023 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:50.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97e3400700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec114260 0x7f97ec10dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:50.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 -- 172.21.15.26:0/2722652023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec114260 msgr2=0x7f97ec10dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec114260 0x7f97ec10dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.299+0000 7f97ebe00700 1 -- 172.21.15.26:0/2722652023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc005040 con 0x7f97ec0656b0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97ebe00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b3b0 tx=0x7f97dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97dc00e040 con 0x7f97ec0656b0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97dc009430 con 0x7f97ec0656b0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec10e280 con 0x7f97ec0656b0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97dc013e90 con 0x7f97ec0656b0 2024-04-16T07:34:50.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.300+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec10e840 con 0x7f97ec0656b0 2024-04-16T07:34:50.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.301+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec10eab0 con 0x7f97ec0656b0 2024-04-16T07:34:50.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.302+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f97dc013640 con 0x7f97ec0656b0 2024-04-16T07:34:50.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.303+0000 7f97e8c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 0x7f97cc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:50.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.303+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f97dc08fcc0 con 0x7f97ec0656b0 2024-04-16T07:34:50.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.304+0000 7f97e3400700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 0x7f97cc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:50.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.304+0000 7f97e3400700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 0x7f97cc0710c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f97d40041d0 tx=0x7f97d4009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:50.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.309+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f97dc095020 con 0x7f97ec0656b0 2024-04-16T07:34:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:50.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:50.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:50.475+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:50.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.585+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f97ec10f370 con 0x7f97ec0656b0 2024-04-16T07:34:50.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.587+0000 7f97e8c00700 1 -- 172.21.15.26:0/2722652023 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1940 (secure 0 0 0) 0x7f97dc05c4f0 con 0x7f97ec0656b0 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout:e36 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:50.587 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 35 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T07:19:38.524328+0000 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-16T07:30:56.852323+0000 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 85 2024-04-16T07:34:50.588 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=34404} 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 34404 members: 34404 2024-04-16T07:34:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kqpyuh{0:34404} state up:active seq 22 join_fscid=1 addr [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.kwjewe{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.26:6828/2163192915,v1:172.21.15.26:6829/2163192915] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.dpfdje{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6824/2443726614,v1:172.21.15.104:6825/2443726614] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:34:50.590 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi104.zwhiwr{-1:44307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.104:6826/1365033812,v1:172.21.15.104:6827/1365033812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 msgr2=0x7f97cc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 0x7f97cc0710c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f97d40041d0 tx=0x7f97d4009040 comp rx=0 tx=0).stop 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 msgr2=0x7f97ec113d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b3b0 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 shutdown_connections 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f97cc06ec10 0x7f97cc0710c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec0656b0 0x7f97ec113d20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 --2- 172.21.15.26:0/2722652023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97ec114260 0x7f97ec10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 >> 172.21.15.26:0/2722652023 conn(0x7f97ec072bc0 msgr2=0x7f97ec0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 shutdown_connections 2024-04-16T07:34:50.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:50.591+0000 7f97f2c00700 1 -- 172.21.15.26:0/2722652023 wait complete. 2024-04-16T07:34:50.592 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 36 2024-04-16T07:34:50.982 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:50 smithi026 ceph-mon[92215]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:50.982 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:50 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2722652023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:34:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:50 smithi104 ceph-mon[81027]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:50 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2722652023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:51.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:51.189+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:51.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:51.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:51.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:51.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:51.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:51.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:51.476+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:51.479 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-16T07:34:51.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:34:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 -- 172.21.15.26:0/1766135919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 msgr2=0x7fd9fc071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/1766135919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc071dd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0048d0 tx=0x7fd9ec009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 -- 172.21.15.26:0/1766135919 shutdown_connections 2024-04-16T07:34:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/1766135919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc10c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/1766135919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc071dd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 -- 172.21.15.26:0/1766135919 >> 172.21.15.26:0/1766135919 conn(0x7fd9fc0f9fc0 msgr2=0x7fd9fc0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 -- 172.21.15.26:0/1766135919 shutdown_connections 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.727+0000 7fd9fbe00640 1 -- 172.21.15.26:0/1766135919 wait complete. 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 Processor -- start 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 -- start start 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc119690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc119d20 con 0x7fd9fc071a00 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.728+0000 7fd9fbe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc1132e0 con 0x7fd9fc072310 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49286/0 (socket says 172.21.15.26:49286) 2024-04-16T07:34:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 -- 172.21.15.26:0/2994662069 learned_addr learned my addr 172.21.15.26:0/2994662069 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9fa000640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc119690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 -- 172.21.15.26:0/2994662069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 msgr2=0x7fd9fc119690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc119690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9faa00640 1 -- 172.21.15.26:0/2994662069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec005040 con 0x7fd9fc071a00 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.729+0000 7fd9fa000640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc119690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9faa00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0047f0 tx=0x7fd9ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9ec0094c0 con 0x7fd9fc071a00 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9ec032e50 con 0x7fd9fc071a00 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9ec0435f0 con 0x7fd9fc071a00 2024-04-16T07:34:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9fc113560 con 0x7fd9fc071a00 2024-04-16T07:34:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.730+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9fc113a20 con 0x7fd9fc071a00 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.731+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9fc073140 con 0x7fd9fc071a00 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.732+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9ec02f070 con 0x7fd9fc071a00 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.733+0000 7fd9f2a00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 0x7fd9e0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.734+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fd9ec0b08b0 con 0x7fd9fc071a00 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.734+0000 7fd9fa000640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 0x7fd9e0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.734+0000 7fd9fa000640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 0x7fd9e0071030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8006fd0 tx=0x7fd9e8008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.738+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd9ec0794b0 con 0x7fd9fc071a00 2024-04-16T07:34:51.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.943+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9fc114850 con 0x7fd9fc071a00 2024-04-16T07:34:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.944+0000 7fd9f2a00640 1 -- 172.21.15.26:0/2994662069 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16975 (secure 0 0 0) 0x7fd9ec07cb70 con 0x7fd9fc071a00 2024-04-16T07:34:51.945 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:51.946 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":85,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:30:53.829972+0000","last_up_change":"2024-04-16T07:29:20.046419+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T07:19:37.442413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T07:19:38.080594+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6803","nonce":214439947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6805","nonce":214439947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6809","nonce":214439947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6807","nonce":214439947}]},"public_addr":"172.21.15.26:6803/214439947","cluster_addr":"172.21.15.26:6805/214439947","heartbeat_back_addr":"172.21.15.26:6809/214439947","heartbeat_front_addr":"172.21.15.26:6807/214439947","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6811","nonce":547286504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6813","nonce":547286504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6817","nonce":547286504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6815","nonce":547286504}]},"public_addr":"172.21.15.26:6811/547286504","cluster_addr":"172.21.15.26:6813/547286504","heartbeat_back_addr":"172.21.15.26:6817/547286504","heartbeat_front_addr":"172.21.15.26:6815/547286504","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6819","nonce":466625097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6821","nonce":466625097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6825","nonce":466625097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6823","nonce":466625097}]},"public_addr":"172.21.15.26:6819/466625097","cluster_addr":"172.21.15.26:6821/466625097","heartbeat_back_addr":"172.21.15.26:6825/466625097","heartbeat_front_addr":"172.21.15.26:6823/466625097","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6801","nonce":1178751206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6803","nonce":1178751206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6807","nonce":1178751206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6805","nonce":1178751206}]},"public_addr":"172.21.15.104:6801/1178751206","cluster_addr":"172.21.15.104:6803/1178751206","heartbeat_back_addr":"172.21.15.104:6807/1178751206","heartbeat_front_addr":"172.21.15.104:6805/1178751206","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6809","nonce":6937815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6811","nonce":6937815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6815","nonce":6937815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6813","nonce":6937815}]},"public_addr":"172.21.15.104:6809/6937815","cluster_addr":"172.21.15.104:6811/6937815","heartbeat_back_addr":"172.21.15.104:6815/6937815","heartbeat_front_addr":"172.21.15.104:6813/6937815","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6817","nonce":2796309483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6819","nonce":2796309483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6823","nonce":2796309483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6821","nonce":2796309483}]},"public_addr":"172.21.15.104:6817/2796309483","cluster_addr":"172.21.15.104:6819/2796309483","heartbeat_back_addr":"172.21.15.104:6823/2796309483","heartbeat_front_addr":"172.21.15.104:6821/2796309483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6824/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/4200135895":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2153800113":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6827/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.26:0/2992109432":"2024-04-17T07:22:24.885942+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/81762164":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3568818565":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/683618636":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/776549524":"2024-04-17T07:24:49.215054+0000","172.21.15.26:6829/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:0/3830655902":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/1883401317":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.104:0/2105197119":"2024-04-17T07:23:44.851156+0000","172.21.15.104:0/1307182004":"2024-04-17T07:23:44.851156+0000","172.21.15.104:6829/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.104:6828/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6828/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:6826/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6827/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.104:6825/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/451733185":"2024-04-17T07:22:24.885942+0000","172.21.15.104:0/3503948742":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/846866607":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/282870970":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/431791760":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6826/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.104:0/2487700588":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/846866607":"2024-04-17T07:24:49.215054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:34:51.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 msgr2=0x7fd9e0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 0x7fd9e0071030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8006fd0 tx=0x7fd9e8008040 comp rx=0 tx=0).stop 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 msgr2=0x7fd9fc119150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0047f0 tx=0x7fd9ec009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 shutdown_connections 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd9e006eb80 0x7fd9e0071030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9fc072310 0x7fd9fc119690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 --2- 172.21.15.26:0/2994662069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9fc071a00 0x7fd9fc119150 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 >> 172.21.15.26:0/2994662069 conn(0x7fd9fc0f9fc0 msgr2=0x7fd9fc0fbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 shutdown_connections 2024-04-16T07:34:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:51.948+0000 7fd9fbe00640 1 -- 172.21.15.26:0/2994662069 wait complete. 2024-04-16T07:34:51.972 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T07:16:23.932743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T07:19:37.442413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T07:19:38.080594+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T07:34:51.972 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T07:34:52.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:52.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:52.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:52.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:52.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:52.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:52.189+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 -- 172.21.15.26:0/518839791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7200102040 msgr2=0x7f7200102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 --2- 172.21.15.26:0/518839791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7200102040 0x7f7200102410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f71f40048d0 tx=0x7f71f4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 -- 172.21.15.26:0/518839791 shutdown_connections 2024-04-16T07:34:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 --2- 172.21.15.26:0/518839791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72001029e0 0x7f7200064400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 --2- 172.21.15.26:0/518839791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7200102040 0x7f7200102410 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 -- 172.21.15.26:0/518839791 >> 172.21.15.26:0/518839791 conn(0x7f7200071df0 msgr2=0x7f72000721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 -- 172.21.15.26:0/518839791 shutdown_connections 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.196+0000 7f7206800640 1 -- 172.21.15.26:0/518839791 wait complete. 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.197+0000 7f7206800640 1 Processor -- start 2024-04-16T07:34:52.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.197+0000 7f7206800640 1 -- start start 2024-04-16T07:34:52.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7206800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7200102040 0x7f7200112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7206800640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7206800640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7200113750 con 0x7f72001029e0 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7206800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720010cda0 con 0x7f7200102040 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49290/0 (socket says 172.21.15.26:49290) 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 -- 172.21.15.26:0/2007785669 learned_addr learned my addr 172.21.15.26:0/2007785669 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 -- 172.21.15.26:0/2007785669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7200102040 msgr2=0x7f7200112c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7200102040 0x7f7200112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.198+0000 7f7204a00640 1 -- 172.21.15.26:0/2007785669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4005040 con 0x7f72001029e0 2024-04-16T07:34:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f7204a00640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f71ec002840 tx=0x7f71ec004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ec007c30 con 0x7f72001029e0 2024-04-16T07:34:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720010d080 con 0x7f72001029e0 2024-04-16T07:34:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71ec00f040 con 0x7f72001029e0 2024-04-16T07:34:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ec009590 con 0x7f72001029e0 2024-04-16T07:34:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.199+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720010d5a0 con 0x7f72001029e0 2024-04-16T07:34:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.200+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72000fd7c0 con 0x7f72001029e0 2024-04-16T07:34:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.201+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f71ec00d6b0 con 0x7f72001029e0 2024-04-16T07:34:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.203+0000 7f71fd600640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 0x7f71e8071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.203+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f71ec08e1d0 con 0x7f72001029e0 2024-04-16T07:34:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.203+0000 7f7205400640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 0x7f71e8071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.204+0000 7f7205400640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 0x7f71e8071030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f71f4003080 tx=0x7f71f4009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.207+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f71ec056e50 con 0x7f72001029e0 2024-04-16T07:34:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.415+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f720004bcc0 con 0x7f72001029e0 2024-04-16T07:34:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.416+0000 7f71fd600640 1 -- 172.21.15.26:0/2007785669 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71ec05a510 con 0x7f72001029e0 2024-04-16T07:34:52.416 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-16T07:34:52.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 msgr2=0x7f71e8071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 0x7f71e8071030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f71f4003080 tx=0x7f71f4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 msgr2=0x7f7200113150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f71ec002840 tx=0x7f71ec004570 comp rx=0 tx=0).stop 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 shutdown_connections 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f71e806eb80 0x7f71e8071030 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72001029e0 0x7f7200113150 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 --2- 172.21.15.26:0/2007785669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7200102040 0x7f7200112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 >> 172.21.15.26:0/2007785669 conn(0x7f7200071df0 msgr2=0x7f7200104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 shutdown_connections 2024-04-16T07:34:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.419+0000 7f7206800640 1 -- 172.21.15.26:0/2007785669 wait complete. 2024-04-16T07:34:52.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 45) v5 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:52.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:52.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 14 ==== client_session(renewcaps seq 45) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:52.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:52.476+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 45) v1 from mds.0 2024-04-16T07:34:52.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 -- 172.21.15.26:0/1921333470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec06fdb0 msgr2=0x7febec070220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 --2- 172.21.15.26:0/1921333470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec06fdb0 0x7febec070220 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7febd40048d0 tx=0x7febd4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 -- 172.21.15.26:0/1921333470 shutdown_connections 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 --2- 172.21.15.26:0/1921333470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec06fdb0 0x7febec070220 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 --2- 172.21.15.26:0/1921333470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec0714a0 0x7febec06f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 -- 172.21.15.26:0/1921333470 >> 172.21.15.26:0/1921333470 conn(0x7febec0f7c20 msgr2=0x7febec0fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 -- 172.21.15.26:0/1921333470 shutdown_connections 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.661+0000 7febeb200640 1 -- 172.21.15.26:0/1921333470 wait complete. 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.662+0000 7febeb200640 1 Processor -- start 2024-04-16T07:34:52.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.662+0000 7febeb200640 1 -- start start 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febeb200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec06fdb0 0x7febec119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febeb200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febeb200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec119c80 con 0x7febec0714a0 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febeb200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec1132d0 con 0x7febec06fdb0 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49306/0 (socket says 172.21.15.26:49306) 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 -- 172.21.15.26:0/2811853629 learned_addr learned my addr 172.21.15.26:0/2811853629 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 -- 172.21.15.26:0/2811853629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec06fdb0 msgr2=0x7febec119140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe9e00640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec06fdb0 0x7febec119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:52.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec06fdb0 0x7febec119140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.663+0000 7febe1400640 1 -- 172.21.15.26:0/2811853629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4005040 con 0x7febec0714a0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febe1400640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7febd40048a0 tx=0x7febd4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd40094c0 con 0x7febec0714a0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec113550 con 0x7febec0714a0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febd403b8f0 con 0x7febec0714a0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd40437b0 con 0x7febec0714a0 2024-04-16T07:34:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.664+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec113a70 con 0x7febec0714a0 2024-04-16T07:34:52.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.665+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febec070fb0 con 0x7febec0714a0 2024-04-16T07:34:52.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.667+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7febd403ba60 con 0x7febec0714a0 2024-04-16T07:34:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.668+0000 7febe2a00640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 0x7febd0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.668+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7febd40b1640 con 0x7febec0714a0 2024-04-16T07:34:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.669+0000 7febe9e00640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 0x7febd0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.669+0000 7febe9e00640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 0x7febd0071020 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7febdc004bc0 tx=0x7febdc009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.673+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7febd407a360 con 0x7febec0714a0 2024-04-16T07:34:52.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:52 smithi026 ceph-mon[92215]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:52.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:52 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2994662069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:34:52.762 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:52 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2007785669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:34:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.878+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7febec1147b0 con 0x7febec0714a0 2024-04-16T07:34:52.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.880+0000 7febe2a00640 1 -- 172.21.15.26:0/2811853629 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7febd407da20 con 0x7febec0714a0 2024-04-16T07:34:52.879 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:34:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.882+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 msgr2=0x7febd0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 0x7febd0071020 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7febdc004bc0 tx=0x7febdc009040 comp rx=0 tx=0).stop 2024-04-16T07:34:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 msgr2=0x7febec119680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7febd40048a0 tx=0x7febd4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 shutdown_connections 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7febd006eb70 0x7febd0071020 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febec0714a0 0x7febec119680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 --2- 172.21.15.26:0/2811853629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febec06fdb0 0x7febec119140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 >> 172.21.15.26:0/2811853629 conn(0x7febec0f7c20 msgr2=0x7febec0f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 shutdown_connections 2024-04-16T07:34:52.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:52.883+0000 7febeb200640 1 -- 172.21.15.26:0/2811853629 wait complete. 2024-04-16T07:34:52.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T07:34:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:52 smithi104 ceph-mon[81027]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:52 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2994662069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:34:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:52 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2007785669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:34:53.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.120+0000 7f2095a00640 1 -- 172.21.15.26:0/2929102886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901086a0 msgr2=0x7f2090108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.120+0000 7f2095a00640 1 --2- 172.21.15.26:0/2929102886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901086a0 0x7f2090108a70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2080009c50 tx=0x7f2080004d10 comp rx=0 tx=0).stop 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 -- 172.21.15.26:0/2929102886 shutdown_connections 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 --2- 172.21.15.26:0/2929102886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901026d0 0x7f2090102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 --2- 172.21.15.26:0/2929102886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901086a0 0x7f2090108a70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 -- 172.21.15.26:0/2929102886 >> 172.21.15.26:0/2929102886 conn(0x7f2090071e70 msgr2=0x7f2090074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 -- 172.21.15.26:0/2929102886 shutdown_connections 2024-04-16T07:34:53.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.121+0000 7f2095a00640 1 -- 172.21.15.26:0/2929102886 wait complete. 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 Processor -- start 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 -- start start 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 0x7f20901196d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2090119d60 con 0x7f20901026d0 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.122+0000 7f2095a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2090113320 con 0x7f20901086a0 2024-04-16T07:34:53.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49332/0 (socket says 172.21.15.26:49332) 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 -- 172.21.15.26:0/1779719622 learned_addr learned my addr 172.21.15.26:0/1779719622 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208f400640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 0x7f20901196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 -- 172.21.15.26:0/1779719622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 msgr2=0x7f20901196d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 0x7f20901196d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 -- 172.21.15.26:0/1779719622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20800098b0 con 0x7f20901026d0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208f400640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 0x7f20901196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.123+0000 7f208fe00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f208002c1e0 tx=0x7f2080004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.124+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f208003a070 con 0x7f20901026d0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.124+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2080009740 con 0x7f20901026d0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.124+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20901135a0 con 0x7f20901026d0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.124+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f208003eb40 con 0x7f20901026d0 2024-04-16T07:34:53.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.124+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2090113ac0 con 0x7f20901026d0 2024-04-16T07:34:53.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.125+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2090103dc0 con 0x7f20901026d0 2024-04-16T07:34:53.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.126+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2080032070 con 0x7f20901026d0 2024-04-16T07:34:53.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.127+0000 7f2087e00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 0x7f2074071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.127+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f20800b0f00 con 0x7f20901026d0 2024-04-16T07:34:53.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.127+0000 7f208f400640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 0x7f2074071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.128+0000 7f208f400640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 0x7f2074071030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f207c006fd0 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:53.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.132+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2080079c70 con 0x7f20901026d0 2024-04-16T07:34:53.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:53.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:53.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:53.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:53.189+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:53.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.336+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2090114890 con 0x7f20901026d0 2024-04-16T07:34:53.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.337+0000 7f2087e00640 1 -- 172.21.15.26:0/1779719622 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2080082e60 con 0x7f20901026d0 2024-04-16T07:34:53.337 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:34:53.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.340+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 msgr2=0x7f2074071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.340+0000 7f2095a00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 0x7f2074071030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f207c006fd0 tx=0x7f207c008040 comp rx=0 tx=0).stop 2024-04-16T07:34:53.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.340+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 msgr2=0x7f2090119190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.340+0000 7f2095a00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f208002c1e0 tx=0x7f2080004d10 comp rx=0 tx=0).stop 2024-04-16T07:34:53.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 shutdown_connections 2024-04-16T07:34:53.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f207406eb80 0x7f2074071030 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20901086a0 0x7f20901196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 --2- 172.21.15.26:0/1779719622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20901026d0 0x7f2090119190 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 >> 172.21.15.26:0/1779719622 conn(0x7f2090071e70 msgr2=0x7f2090073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 shutdown_connections 2024-04-16T07:34:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.341+0000 7f2095a00640 1 -- 172.21.15.26:0/1779719622 wait complete. 2024-04-16T07:34:53.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:53.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:53.476+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 -- 172.21.15.26:0/2442757379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 msgr2=0x7f8994107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 --2- 172.21.15.26:0/2442757379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f8994107930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f89840048d0 tx=0x7f8984009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 -- 172.21.15.26:0/2442757379 shutdown_connections 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 --2- 172.21.15.26:0/2442757379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f8994102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 --2- 172.21.15.26:0/2442757379 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f8994107930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 -- 172.21.15.26:0/2442757379 >> 172.21.15.26:0/2442757379 conn(0x7f8994071e50 msgr2=0x7f8994074260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 -- 172.21.15.26:0/2442757379 shutdown_connections 2024-04-16T07:34:53.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.587+0000 7f8993200640 1 -- 172.21.15.26:0/2442757379 wait complete. 2024-04-16T07:34:53.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 Processor -- start 2024-04-16T07:34:53.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 -- start start 2024-04-16T07:34:53.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f89941190b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994119bf0 con 0x7f8994107560 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.588+0000 7f8993200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994119d60 con 0x7f8994102630 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49360/0 (socket says 172.21.15.26:49360) 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 -- 172.21.15.26:0/293719776 learned_addr learned my addr 172.21.15.26:0/293719776 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f8991e00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f89941190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 -- 172.21.15.26:0/293719776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 msgr2=0x7f89941190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f89941190b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f898be00640 1 -- 172.21.15.26:0/293719776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984005040 con 0x7f8994107560 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.589+0000 7f8991e00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f89941190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f898be00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f897c002a50 tx=0x7f897c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c007c30 con 0x7f8994107560 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f897c00f040 con 0x7f8994107560 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c00db50 con 0x7f8994107560 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89941133f0 con 0x7f8994107560 2024-04-16T07:34:53.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.590+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8994113910 con 0x7f8994107560 2024-04-16T07:34:53.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.591+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8994103d20 con 0x7f8994107560 2024-04-16T07:34:53.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.592+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f897c00dcf0 con 0x7f8994107560 2024-04-16T07:34:53.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.593+0000 7f898a000640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 0x7f8978071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:53.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.593+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f897c08e290 con 0x7f8994107560 2024-04-16T07:34:53.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.594+0000 7f8991e00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 0x7f8978071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:53.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.594+0000 7f8991e00640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 0x7f8978071030 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8984000c00 tx=0x7f8984009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:53.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.598+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f897c056f10 con 0x7f8994107560 2024-04-16T07:34:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.853+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8994113b80 con 0x7f8994107560 2024-04-16T07:34:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.854+0000 7f898a000640 1 -- 172.21.15.26:0/293719776 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5213 (secure 0 0 0) 0x7f897c05a5d0 con 0x7f8994107560 2024-04-16T07:34:53.854 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:53.854 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":44307,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6827/1365033812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1365033812},{"type":"v1","addr":"172.21.15.104:6827","nonce":1365033812}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:56.852323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34404":{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":0,"incarnation":32,"state":"up:active","state_seq":22,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34404,"qdb_cluster":[34404]},"id":1}]} 2024-04-16T07:34:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 36 2024-04-16T07:34:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 msgr2=0x7f8978071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 0x7f8978071030 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8984000c00 tx=0x7f8984009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 msgr2=0x7f89941195f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f897c002a50 tx=0x7f897c004600 comp rx=0 tx=0).stop 2024-04-16T07:34:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 shutdown_connections 2024-04-16T07:34:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f897806eb80 0x7f8978071030 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8994107560 0x7f89941195f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 --2- 172.21.15.26:0/293719776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8994102630 0x7f89941190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.857+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 >> 172.21.15.26:0/293719776 conn(0x7f8994071e50 msgr2=0x7f8994074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.858+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 shutdown_connections 2024-04-16T07:34:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:53.858+0000 7f8993200640 1 -- 172.21.15.26:0/293719776 wait complete. 2024-04-16T07:34:53.874 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-04-16T07:34:53.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-04-16T07:34:53.959 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:53 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2811853629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:34:53.959 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:53 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1779719622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:34:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:53 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2811853629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:34:54.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:53 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1779719622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.095+0000 7f5843200640 1 -- 172.21.15.26:0/648667677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58441029e0 msgr2=0x7f5844064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.095+0000 7f5843200640 1 --2- 172.21.15.26:0/648667677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58441029e0 0x7f5844064380 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f582c009c50 tx=0x7f582c004d10 comp rx=0 tx=0).stop 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.095+0000 7f5843200640 1 -- 172.21.15.26:0/648667677 shutdown_connections 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.095+0000 7f5843200640 1 --2- 172.21.15.26:0/648667677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58441029e0 0x7f5844064380 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.095+0000 7f5843200640 1 --2- 172.21.15.26:0/648667677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5844102040 0x7f5844102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.096+0000 7f5843200640 1 -- 172.21.15.26:0/648667677 >> 172.21.15.26:0/648667677 conn(0x7f5844071010 msgr2=0x7f5844071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:54.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.096+0000 7f5843200640 1 -- 172.21.15.26:0/648667677 shutdown_connections 2024-04-16T07:34:54.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.096+0000 7f5843200640 1 -- 172.21.15.26:0/648667677 wait complete. 2024-04-16T07:34:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.096+0000 7f5843200640 1 Processor -- start 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5843200640 1 -- start start 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5843200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5843200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58441029e0 0x7f58441152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5843200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441158e0 con 0x7f5844102040 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5843200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584410ef80 con 0x7f58441029e0 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5841e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5841e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49390/0 (socket says 172.21.15.26:49390) 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.097+0000 7f5841e00640 1 -- 172.21.15.26:0/165905730 learned_addr learned my addr 172.21.15.26:0/165905730 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f5841400640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58441029e0 0x7f58441152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f5841e00640 1 -- 172.21.15.26:0/165905730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58441029e0 msgr2=0x7f58441152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f5841e00640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58441029e0 0x7f58441152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f5841e00640 1 -- 172.21.15.26:0/165905730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0098b0 con 0x7f5844102040 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f5841e00640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5834002a50 tx=0x7f5834004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.098+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5834007c30 con 0x7f5844102040 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.099+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f583400f040 con 0x7f5844102040 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.099+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583400db50 con 0x7f5844102040 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.099+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584410f260 con 0x7f5844102040 2024-04-16T07:34:54.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.099+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584410f750 con 0x7f5844102040 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.100+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58440fd7e0 con 0x7f5844102040 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.101+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f583400dcf0 con 0x7f5844102040 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.102+0000 7f583a000640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 0x7f5824071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.103+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f583408e290 con 0x7f5844102040 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.103+0000 7f5841400640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 0x7f5824071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.104+0000 7f5841400640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 0x7f5824071020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f582c009c20 tx=0x7f582c02e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.108+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5834056f10 con 0x7f5844102040 2024-04-16T07:34:54.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:54.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:54.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:54.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:54.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:54.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:54.189+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:54.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.349+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f5844110440 con 0x7f5844102040 2024-04-16T07:34:54.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.350+0000 7f583a000640 1 -- 172.21.15.26:0/165905730 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v36) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f583405a5d0 con 0x7f5844102040 2024-04-16T07:34:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:54.352 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14480,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/4271222482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4271222482},{"type":"v1","addr":"172.21.15.26:6827","nonce":4271222482}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:19:59.548919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi026.kwjewe","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.26:6829/1284848687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1284848687},{"type":"v1","addr":"172.21.15.26:6829","nonce":1284848687}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:54.353 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 16 2024-04-16T07:34:54.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 msgr2=0x7f5824071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 0x7f5824071020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f582c009c20 tx=0x7f582c02e040 comp rx=0 tx=0).stop 2024-04-16T07:34:54.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 msgr2=0x7f5844114da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5834002a50 tx=0x7f5834004600 comp rx=0 tx=0).stop 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 shutdown_connections 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f582406eb70 0x7f5824071020 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58441029e0 0x7f58441152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 --2- 172.21.15.26:0/165905730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5844102040 0x7f5844114da0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 >> 172.21.15.26:0/165905730 conn(0x7f5844071010 msgr2=0x7f58440fa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 shutdown_connections 2024-04-16T07:34:54.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.353+0000 7f5843200640 1 -- 172.21.15.26:0/165905730 wait complete. 2024-04-16T07:34:54.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-16T07:34:54.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:54.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:54.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:54.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:54.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:54.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:54.476+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 -- 172.21.15.26:0/248109334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 msgr2=0x7f2744104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 --2- 172.21.15.26:0/248109334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744104600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f27340048d0 tx=0x7f2734009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 -- 172.21.15.26:0/248109334 shutdown_connections 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 --2- 172.21.15.26:0/248109334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f27440fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 --2- 172.21.15.26:0/248109334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744104600 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 -- 172.21.15.26:0/248109334 >> 172.21.15.26:0/248109334 conn(0x7f27440f9fc0 msgr2=0x7f27440fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:54.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.579+0000 7f2749a00640 1 -- 172.21.15.26:0/248109334 shutdown_connections 2024-04-16T07:34:54.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.580+0000 7f2749a00640 1 -- 172.21.15.26:0/248109334 wait complete. 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.580+0000 7f2749a00640 1 Processor -- start 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.580+0000 7f2749a00640 1 -- start start 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2749a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2749a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f2744115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2749a00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2744115830 con 0x7f2744104230 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2749a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274410ee80 con 0x7f2744104bd0 2024-04-16T07:34:54.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2743e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2743e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49416/0 (socket says 172.21.15.26:49416) 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2743e00640 1 -- 172.21.15.26:0/3652036226 learned_addr learned my addr 172.21.15.26:0/3652036226 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.581+0000 7f2743400640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f2744115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2743e00640 1 -- 172.21.15.26:0/3652036226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 msgr2=0x7f2744115230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2743e00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f2744115230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2743e00640 1 -- 172.21.15.26:0/3652036226 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734005040 con 0x7f2744104230 2024-04-16T07:34:54.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2743400640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f2744115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T07:34:54.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2743e00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2734004930 tx=0x7f2734009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27340094c0 con 0x7f2744104230 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2734032e50 con 0x7f2744104230 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.582+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274410f100 con 0x7f2744104230 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.583+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27340433b0 con 0x7f2744104230 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.583+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274410f5c0 con 0x7f2744104230 2024-04-16T07:34:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.584+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27440ff9a0 con 0x7f2744104230 2024-04-16T07:34:54.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.585+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f273402f070 con 0x7f2744104230 2024-04-16T07:34:54.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.586+0000 7f273be00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 0x7f2728071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:54.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.586+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f27340b0870 con 0x7f2744104230 2024-04-16T07:34:54.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.586+0000 7f2743400640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 0x7f2728071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:54.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.587+0000 7f2743400640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 0x7f2728071020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730009e90 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:54.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.592+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2734079510 con 0x7f2744104230 2024-04-16T07:34:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.834+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f27441103f0 con 0x7f2744104230 2024-04-16T07:34:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.835+0000 7f273be00640 1 -- 172.21.15.26:0/3652036226 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v36) v1 ==== 107+0+4081 (secure 0 0 0) 0x7f273407cbd0 con 0x7f2744104230 2024-04-16T07:34:54.835 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:54.835 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:19:59.548919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi026.kwjewe","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.26:6829/1284848687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1284848687},{"type":"v1","addr":"172.21.15.26:6829","nonce":1284848687}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 17 2024-04-16T07:34:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 msgr2=0x7f2728071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 0x7f2728071020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730009e90 comp rx=0 tx=0).stop 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 msgr2=0x7f2744114cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2734004930 tx=0x7f2734009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 shutdown_connections 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f272806eb70 0x7f2728071020 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2744104bd0 0x7f2744115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 --2- 172.21.15.26:0/3652036226 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2744104230 0x7f2744114cf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 >> 172.21.15.26:0/3652036226 conn(0x7f27440f9fc0 msgr2=0x7f27440fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 shutdown_connections 2024-04-16T07:34:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:54.838+0000 7f2749a00640 1 -- 172.21.15.26:0/3652036226 wait complete. 2024-04-16T07:34:54.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-16T07:34:54.935 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:54 smithi026 ceph-mon[92215]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:54.935 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:54 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/293719776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:34:54.935 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:54 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/165905730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 -- 172.21.15.26:0/2988011275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae04100f10 msgr2=0x7fae04064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 --2- 172.21.15.26:0/2988011275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae04100f10 0x7fae04064380 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fadec0048d0 tx=0x7fadec009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 -- 172.21.15.26:0/2988011275 shutdown_connections 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 --2- 172.21.15.26:0/2988011275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae040648c0 0x7fae04064d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 --2- 172.21.15.26:0/2988011275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae04100f10 0x7fae04064380 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 -- 172.21.15.26:0/2988011275 >> 172.21.15.26:0/2988011275 conn(0x7fae04071010 msgr2=0x7fae04071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 -- 172.21.15.26:0/2988011275 shutdown_connections 2024-04-16T07:34:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.066+0000 7fae09800640 1 -- 172.21.15.26:0/2988011275 wait complete. 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.067+0000 7fae09800640 1 Processor -- start 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.067+0000 7fae09800640 1 -- start start 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae09800640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae09800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae04100f10 0x7fae04119630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae09800640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04119c30 con 0x7fae040648c0 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae09800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04113280 con 0x7fae04100f10 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49428/0 (socket says 172.21.15.26:49428) 2024-04-16T07:34:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 -- 172.21.15.26:0/266125745 learned_addr learned my addr 172.21.15.26:0/266125745 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 -- 172.21.15.26:0/266125745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae04100f10 msgr2=0x7fae04119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae04100f10 0x7fae04119630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.068+0000 7fae03e00640 1 -- 172.21.15.26:0/266125745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec005040 con 0x7fae040648c0 2024-04-16T07:34:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fae03e00640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fadec004930 tx=0x7fadec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadec0094c0 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadec032e50 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae04113500 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadec0433b0 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.069+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae041139c0 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.071+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040fb5d0 con 0x7fae040648c0 2024-04-16T07:34:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.071+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fadec02f070 con 0x7fae040648c0 2024-04-16T07:34:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.073+0000 7fadfbe00640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 0x7fade8071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.073+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fadec0b08d0 con 0x7fae040648c0 2024-04-16T07:34:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.073+0000 7fae03400640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 0x7fade8071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.074+0000 7fae03400640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 0x7fade8071020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae041149b0 tx=0x7fadf4004f70 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:55.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.078+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fadec079510 con 0x7fae040648c0 2024-04-16T07:34:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:54 smithi104 ceph-mon[81027]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:54 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/293719776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T07:34:55.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:54 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/165905730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T07:34:55.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:55.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:55.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:55.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:55.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:55.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:55.189+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:55.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.320+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7fae0410f7f0 con 0x7fae040648c0 2024-04-16T07:34:55.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.322+0000 7fadfbe00640 1 -- 172.21.15.26:0/266125745 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v36) v1 ==== 107+0+4928 (secure 0 0 0) 0x7fadec07cbd0 con 0x7fae040648c0 2024-04-16T07:34:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:55.322 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:19:59.548919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi026.kwjewe","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.26:6829/1284848687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1284848687},{"type":"v1","addr":"172.21.15.26:6829","nonce":1284848687}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:55.322 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2024-04-16T07:34:55.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 msgr2=0x7fade8071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 0x7fade8071020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae041149b0 tx=0x7fadf4004f70 comp rx=0 tx=0).stop 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 msgr2=0x7fae041190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fadec004930 tx=0x7fadec009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 shutdown_connections 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fade806eb70 0x7fade8071020 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae04100f10 0x7fae04119630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 --2- 172.21.15.26:0/266125745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae040648c0 0x7fae041190f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.324+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 >> 172.21.15.26:0/266125745 conn(0x7fae04071010 msgr2=0x7fae040f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:55.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.325+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 shutdown_connections 2024-04-16T07:34:55.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.325+0000 7fae09800640 1 -- 172.21.15.26:0/266125745 wait complete. 2024-04-16T07:34:55.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-16T07:34:55.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:55.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:55.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:55.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:55.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:55.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:55.477+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:55.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 -- 172.21.15.26:0/3965553355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed4104d60 msgr2=0x7f7ed40fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 --2- 172.21.15.26:0/3965553355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed4104d60 0x7f7ed40fe3f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0048d0 tx=0x7f7ebc009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 -- 172.21.15.26:0/3965553355 shutdown_connections 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 --2- 172.21.15.26:0/3965553355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed4104d60 0x7f7ed40fe3f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 --2- 172.21.15.26:0/3965553355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ed41043c0 0x7f7ed4104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 -- 172.21.15.26:0/3965553355 >> 172.21.15.26:0/3965553355 conn(0x7f7ed40f9fc0 msgr2=0x7f7ed40fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.549+0000 7f7eda400640 1 -- 172.21.15.26:0/3965553355 shutdown_connections 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.550+0000 7f7eda400640 1 -- 172.21.15.26:0/3965553355 wait complete. 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.550+0000 7f7eda400640 1 Processor -- start 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.550+0000 7f7eda400640 1 -- start start 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7eda400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7eda400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ed4104d60 0x7f7ed4115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7eda400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed4115a00 con 0x7f7ed41043c0 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7eda400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed410f050 con 0x7f7ed4104d60 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49456/0 (socket says 172.21.15.26:49456) 2024-04-16T07:34:55.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 -- 172.21.15.26:0/1637242846 learned_addr learned my addr 172.21.15.26:0/1637242846 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 -- 172.21.15.26:0/1637242846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ed4104d60 msgr2=0x7f7ed4115400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ed4104d60 0x7f7ed4115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.551+0000 7f7ed3400640 1 -- 172.21.15.26:0/1637242846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc005040 con 0x7f7ed41043c0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7ed3400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400db00 tx=0x7f7ec4004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec4017070 con 0x7f7ed41043c0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ec400f040 con 0x7f7ed41043c0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed410f330 con 0x7f7ed41043c0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec4003ae0 con 0x7f7ed41043c0 2024-04-16T07:34:55.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.552+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed410f850 con 0x7f7ed41043c0 2024-04-16T07:34:55.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.553+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed40ffb30 con 0x7f7ed41043c0 2024-04-16T07:34:55.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.554+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ec4003c40 con 0x7f7ed41043c0 2024-04-16T07:34:55.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.555+0000 7f7ecbe00640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 0x7f7eb8071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.555+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f7ec408ead0 con 0x7f7ed41043c0 2024-04-16T07:34:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.556+0000 7f7ecaa00640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 0x7f7eb8071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.556+0000 7f7ecaa00640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 0x7f7eb8071020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0043d0 tx=0x7f7ebc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:55.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.560+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ec4057750 con 0x7f7ed41043c0 2024-04-16T07:34:55.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.810+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f7ed4064380 con 0x7f7ed41043c0 2024-04-16T07:34:55.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.813+0000 7f7ecbe00640 1 -- 172.21.15.26:0/1637242846 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v36) v1 ==== 107+0+4138 (secure 0 0 0) 0x7f7ec405ae10 con 0x7f7ed41043c0 2024-04-16T07:34:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:55.812 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:00.990936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:55.813 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 19 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 msgr2=0x7f7eb8071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 0x7f7eb8071020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0043d0 tx=0x7f7ebc009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 msgr2=0x7f7ed4114ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400db00 tx=0x7f7ec4004e60 comp rx=0 tx=0).stop 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 shutdown_connections 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7eb806eb70 0x7f7eb8071020 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ed4104d60 0x7f7ed4115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 --2- 172.21.15.26:0/1637242846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ed41043c0 0x7f7ed4114ec0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 >> 172.21.15.26:0/1637242846 conn(0x7f7ed40f9fc0 msgr2=0x7f7ed41023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 shutdown_connections 2024-04-16T07:34:55.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:55.816+0000 7f7eda400640 1 -- 172.21.15.26:0/1637242846 wait complete. 2024-04-16T07:34:55.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-16T07:34:55.911 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:55 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3652036226' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T07:34:55.912 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:55 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/266125745' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 -- 172.21.15.26:0/4065061901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104230 msgr2=0x7f2c80104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 --2- 172.21.15.26:0/4065061901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104230 0x7f2c80104600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c740048d0 tx=0x7f2c74009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 -- 172.21.15.26:0/4065061901 shutdown_connections 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 --2- 172.21.15.26:0/4065061901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104bd0 0x7f2c800fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 --2- 172.21.15.26:0/4065061901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104230 0x7f2c80104600 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.045+0000 7f2c87000640 1 -- 172.21.15.26:0/4065061901 >> 172.21.15.26:0/4065061901 conn(0x7f2c800f9fc0 msgr2=0x7f2c800fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.046+0000 7f2c87000640 1 -- 172.21.15.26:0/4065061901 shutdown_connections 2024-04-16T07:34:56.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.046+0000 7f2c87000640 1 -- 172.21.15.26:0/4065061901 wait complete. 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.046+0000 7f2c87000640 1 Processor -- start 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c87000640 1 -- start start 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c87000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104230 0x7f2c80114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c87000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c87000640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80115830 con 0x7f2c80104bd0 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c87000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8010ee80 con 0x7f2c80104230 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c85200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c85200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49474/0 (socket says 172.21.15.26:49474) 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.047+0000 7f2c85200640 1 -- 172.21.15.26:0/2439972470 learned_addr learned my addr 172.21.15.26:0/2439972470 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c85c00640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104230 0x7f2c80114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c85200640 1 -- 172.21.15.26:0/2439972470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104230 msgr2=0x7f2c80114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c85200640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104230 0x7f2c80114cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c85200640 1 -- 172.21.15.26:0/2439972470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c74005040 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c85200640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2c70002a50 tx=0x7f2c70004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c70007c30 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c7000f040 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8010f160 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.048+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c7000dad0 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.049+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8010f680 con 0x7f2c80104bd0 2024-04-16T07:34:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.050+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c800ff9a0 con 0x7f2c80104bd0 2024-04-16T07:34:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.051+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c7000dd40 con 0x7f2c80104bd0 2024-04-16T07:34:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.052+0000 7f2c7d600640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 0x7f2c68071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.052+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f2c7008e2d0 con 0x7f2c80104bd0 2024-04-16T07:34:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.052+0000 7f2c85c00640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 0x7f2c68071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.053+0000 7f2c85c00640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 0x7f2c68071020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c740048a0 tx=0x7f2c74009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:56.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.057+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2c70056ed0 con 0x7f2c80104bd0 2024-04-16T07:34:56.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:55 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3652036226' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T07:34:56.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:55 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/266125745' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T07:34:56.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.189+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:56.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.189+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:56.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.189+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:56.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.189+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:56.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.190+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:56.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:56.190+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.305+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f2c8010b3f0 con 0x7f2c80104bd0 2024-04-16T07:34:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.306+0000 7f2c7d600640 1 -- 172.21.15.26:0/2439972470 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v36) v1 ==== 107+0+4149 (secure 0 0 0) 0x7f2c70061030 con 0x7f2c80104bd0 2024-04-16T07:34:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:56.306 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:01.004418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":0,"incarnation":20,"state":"up:replay","state_seq":4,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 20 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.309+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 msgr2=0x7f2c68071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.309+0000 7f2c87000640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 0x7f2c68071020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c740048a0 tx=0x7f2c74009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 msgr2=0x7f2c80115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2c70002a50 tx=0x7f2c70004600 comp rx=0 tx=0).stop 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 shutdown_connections 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f2c6806eb70 0x7f2c68071020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c80104bd0 0x7f2c80115230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 --2- 172.21.15.26:0/2439972470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c80104230 0x7f2c80114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 >> 172.21.15.26:0/2439972470 conn(0x7f2c800f9fc0 msgr2=0x7f2c800fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 shutdown_connections 2024-04-16T07:34:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.310+0000 7f2c87000640 1 -- 172.21.15.26:0/2439972470 wait complete. 2024-04-16T07:34:56.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-16T07:34:56.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:56.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:56.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:56.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:56.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:56.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:56.477+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 -- 172.21.15.26:0/2967814271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 msgr2=0x7fa5a4108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 --2- 172.21.15.26:0/2967814271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a4108c30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa5980048d0 tx=0x7fa598009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 -- 172.21.15.26:0/2967814271 shutdown_connections 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 --2- 172.21.15.26:0/2967814271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a4102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 --2- 172.21.15.26:0/2967814271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a4108c30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 -- 172.21.15.26:0/2967814271 >> 172.21.15.26:0/2967814271 conn(0x7fa5a4072050 msgr2=0x7fa5a4074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 -- 172.21.15.26:0/2967814271 shutdown_connections 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.549+0000 7fa5ac400640 1 -- 172.21.15.26:0/2967814271 wait complete. 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 Processor -- start 2024-04-16T07:34:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 -- start start 2024-04-16T07:34:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a41192a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a4119de0 con 0x7fa5a4108860 2024-04-16T07:34:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.550+0000 7fa5ac400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a4119f50 con 0x7fa5a4102840 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49488/0 (socket says 172.21.15.26:49488) 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 -- 172.21.15.26:0/4172049658 learned_addr learned my addr 172.21.15.26:0/4172049658 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9a00640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a41192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 -- 172.21.15.26:0/4172049658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 msgr2=0x7fa5a41192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a41192a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 -- 172.21.15.26:0/4172049658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa598005040 con 0x7fa5a4108860 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9a00640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a41192a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.551+0000 7fa5a9000640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa594002a50 tx=0x7fa594004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.552+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa594007c30 con 0x7fa5a4108860 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.552+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa59400f040 con 0x7fa5a4108860 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.552+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa59400db50 con 0x7fa5a4108860 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.552+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a41135e0 con 0x7fa5a4108860 2024-04-16T07:34:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.552+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a4113b00 con 0x7fa5a4108860 2024-04-16T07:34:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.553+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a4103f80 con 0x7fa5a4108860 2024-04-16T07:34:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.554+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa59400dcf0 con 0x7fa5a4108860 2024-04-16T07:34:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.555+0000 7fa5a1600640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 0x7fa58c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.555+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa59408e1f0 con 0x7fa5a4108860 2024-04-16T07:34:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.555+0000 7fa5a9a00640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 0x7fa58c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.556+0000 7fa5a9a00640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 0x7fa58c071020 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa598009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:56.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.560+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa594056f10 con 0x7fa5a4108860 2024-04-16T07:34:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.809+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fa5a4113d70 con 0x7fa5a4108860 2024-04-16T07:34:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.810+0000 7fa5a1600640 1 -- 172.21.15.26:0/4172049658 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v36) v1 ==== 107+0+4154 (secure 0 0 0) 0x7fa59405a5d0 con 0x7fa5a4108860 2024-04-16T07:34:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:56.811 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:01.031400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":155,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:56.811 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.813+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 msgr2=0x7fa58c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.813+0000 7fa5ac400640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 0x7fa58c071020 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa598009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.813+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 msgr2=0x7fa5a41197e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.813+0000 7fa5ac400640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa594002a50 tx=0x7fa594004600 comp rx=0 tx=0).stop 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 shutdown_connections 2024-04-16T07:34:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa58c06eb70 0x7fa58c071020 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5a4108860 0x7fa5a41197e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 --2- 172.21.15.26:0/4172049658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a4102840 0x7fa5a41192a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 >> 172.21.15.26:0/4172049658 conn(0x7fa5a4072050 msgr2=0x7fa5a4073fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 shutdown_connections 2024-04-16T07:34:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:56.814+0000 7fa5ac400640 1 -- 172.21.15.26:0/4172049658 wait complete. 2024-04-16T07:34:56.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-16T07:34:56.922 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:56 smithi026 ceph-mon[92215]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:56.922 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:56 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1637242846' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T07:34:56.922 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:56 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2439972470' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 -- 172.21.15.26:0/2982154754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104210 msgr2=0x7f08301045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 --2- 172.21.15.26:0/2982154754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104210 0x7f08301045e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f08200048d0 tx=0x7f0820009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 -- 172.21.15.26:0/2982154754 shutdown_connections 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 --2- 172.21.15.26:0/2982154754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0830104bb0 0x7f08300fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 --2- 172.21.15.26:0/2982154754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104210 0x7f08301045e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 -- 172.21.15.26:0/2982154754 >> 172.21.15.26:0/2982154754 conn(0x7f08300f9fc0 msgr2=0x7f08300fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 -- 172.21.15.26:0/2982154754 shutdown_connections 2024-04-16T07:34:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.052+0000 7f082fe00640 1 -- 172.21.15.26:0/2982154754 wait complete. 2024-04-16T07:34:57.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.053+0000 7f082fe00640 1 Processor -- start 2024-04-16T07:34:57.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.053+0000 7f082fe00640 1 -- start start 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f082fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0830104210 0x7f0830114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f082fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f082fe00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0830115860 con 0x7f0830104bb0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f082fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083010eeb0 con 0x7f0830104210 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f0827e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f0827e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49504/0 (socket says 172.21.15.26:49504) 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.054+0000 7f0827e00640 1 -- 172.21.15.26:0/3001194773 learned_addr learned my addr 172.21.15.26:0/3001194773 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0827e00640 1 -- 172.21.15.26:0/3001194773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0830104210 msgr2=0x7f0830114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0827e00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0830104210 0x7f0830114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0827e00640 1 -- 172.21.15.26:0/3001194773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0820005040 con 0x7f0830104bb0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0827e00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0818002a50 tx=0x7f0818004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0818007c30 con 0x7f0830104bb0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f081800f040 con 0x7f0830104bb0 2024-04-16T07:34:57.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083010f190 con 0x7f0830104bb0 2024-04-16T07:34:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081800db50 con 0x7f0830104bb0 2024-04-16T07:34:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.055+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083010f6b0 con 0x7f0830104bb0 2024-04-16T07:34:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.056+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08300ff980 con 0x7f0830104bb0 2024-04-16T07:34:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.057+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f081801c400 con 0x7f0830104bb0 2024-04-16T07:34:57.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.058+0000 7f0826a00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 0x7f0814071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.058+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f081808e340 con 0x7f0830104bb0 2024-04-16T07:34:57.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.058+0000 7f082ea00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 0x7f0814071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:57.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.059+0000 7f082ea00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 0x7f0814071020 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08200043d0 tx=0x7f0820009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:57.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.063+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0818056f40 con 0x7f0830104bb0 2024-04-16T07:34:57.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:56 smithi104 ceph-mon[81027]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:57.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:56 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1637242846' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T07:34:57.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:56 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2439972470' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T07:34:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:57.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:57.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:57.190+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.304+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f083010b3a0 con 0x7f0830104bb0 2024-04-16T07:34:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.305+0000 7f0826a00640 1 -- 172.21.15.26:0/3001194773 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v36) v1 ==== 107+0+4151 (secure 0 0 0) 0x7f081805a600 con 0x7f0830104bb0 2024-04-16T07:34:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:57.305 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:02.025064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":156,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 22 2024-04-16T07:34:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 msgr2=0x7f0814071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 0x7f0814071020 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08200043d0 tx=0x7f0820009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 msgr2=0x7f0830115260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0818002a50 tx=0x7f0818004600 comp rx=0 tx=0).stop 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 shutdown_connections 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f081406eb70 0x7f0814071020 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0830104bb0 0x7f0830115260 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 --2- 172.21.15.26:0/3001194773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0830104210 0x7f0830114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 >> 172.21.15.26:0/3001194773 conn(0x7f08300f9fc0 msgr2=0x7f08300fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 shutdown_connections 2024-04-16T07:34:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.308+0000 7f082fe00640 1 -- 172.21.15.26:0/3001194773 wait complete. 2024-04-16T07:34:57.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:57.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:57.477+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 -- 172.21.15.26:0/3599704037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528101f90 msgr2=0x7f5528102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 --2- 172.21.15.26:0/3599704037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528101f90 0x7f5528102360 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f55100048d0 tx=0x7f5510009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 -- 172.21.15.26:0/3599704037 shutdown_connections 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 --2- 172.21.15.26:0/3599704037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528102930 0x7f55280fbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 --2- 172.21.15.26:0/3599704037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528101f90 0x7f5528102360 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 -- 172.21.15.26:0/3599704037 >> 172.21.15.26:0/3599704037 conn(0x7f55280f7b60 msgr2=0x7f55280f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 -- 172.21.15.26:0/3599704037 shutdown_connections 2024-04-16T07:34:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.542+0000 7f5527200640 1 -- 172.21.15.26:0/3599704037 wait complete. 2024-04-16T07:34:57.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 Processor -- start 2024-04-16T07:34:57.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 -- start start 2024-04-16T07:34:57.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528101f90 0x7f552806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55280702e0 con 0x7f5528102930 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.543+0000 7f5527200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528070450 con 0x7f5528101f90 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49526/0 (socket says 172.21.15.26:49526) 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 -- 172.21.15.26:0/3043694821 learned_addr learned my addr 172.21.15.26:0/3043694821 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525e00640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528101f90 0x7f552806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 -- 172.21.15.26:0/3043694821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528101f90 msgr2=0x7f552806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528101f90 0x7f552806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 -- 172.21.15.26:0/3043694821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5510005040 con 0x7f5528102930 2024-04-16T07:34:57.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.544+0000 7f5525400640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f551800bbb0 tx=0x7f5518009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.545+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f551800c940 con 0x7f5528102930 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.545+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55180092e0 con 0x7f5528102930 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.545+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5518007640 con 0x7f5528102930 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.545+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5528119040 con 0x7f5528102930 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.545+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5528119480 con 0x7f5528102930 2024-04-16T07:34:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.546+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55280fd700 con 0x7f5528102930 2024-04-16T07:34:57.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.547+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5518012070 con 0x7f5528102930 2024-04-16T07:34:57.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.548+0000 7f551e000640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 0x7f550c071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:57.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.548+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f551808e960 con 0x7f5528102930 2024-04-16T07:34:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.548+0000 7f5525e00640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 0x7f550c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.549+0000 7f5525e00640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 0x7f550c071030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55100048a0 tx=0x7f5510009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.553+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f55180575e0 con 0x7f5528102930 2024-04-16T07:34:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.797+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f55281091e0 con 0x7f5528102930 2024-04-16T07:34:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.798+0000 7f551e000640 1 -- 172.21.15.26:0/3043694821 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v36) v1 ==== 107+0+4160 (secure 0 0 0) 0x7f551805aca0 con 0x7f5528102930 2024-04-16T07:34:57.797 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:57.797 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:04.031949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":0,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24287,"qdb_cluster":[24287]},"id":1}]} 2024-04-16T07:34:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 23 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 msgr2=0x7f550c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 0x7f550c071030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55100048a0 tx=0x7f5510009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 msgr2=0x7f552806fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f551800bbb0 tx=0x7f5518009840 comp rx=0 tx=0).stop 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 shutdown_connections 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f550c06eb80 0x7f550c071030 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5528102930 0x7f552806fd10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 --2- 172.21.15.26:0/3043694821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5528101f90 0x7f552806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:57.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.800+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 >> 172.21.15.26:0/3043694821 conn(0x7f55280f7b60 msgr2=0x7f55280f8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:57.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.801+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 shutdown_connections 2024-04-16T07:34:57.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:57.801+0000 7f5527200640 1 -- 172.21.15.26:0/3043694821 wait complete. 2024-04-16T07:34:57.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-16T07:34:58.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:57 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4172049658' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T07:34:58.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:57 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3001194773' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T07:34:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.027+0000 7f9685a00640 1 -- 172.21.15.26:0/679751650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680064380 msgr2=0x7f96800647b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 --2- 172.21.15.26:0/679751650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680064380 0x7f96800647b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f96680048d0 tx=0x7f9668009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 -- 172.21.15.26:0/679751650 shutdown_connections 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 --2- 172.21.15.26:0/679751650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680064380 0x7f96800647b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 --2- 172.21.15.26:0/679751650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680102240 0x7f9680102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 -- 172.21.15.26:0/679751650 >> 172.21.15.26:0/679751650 conn(0x7f96800711c0 msgr2=0x7f96800715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 -- 172.21.15.26:0/679751650 shutdown_connections 2024-04-16T07:34:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.028+0000 7f9685a00640 1 -- 172.21.15.26:0/679751650 wait complete. 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 Processor -- start 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 -- start start 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 0x7f9680114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96801159d0 con 0x7f9680102240 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.029+0000 7f9685a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968010f020 con 0x7f9680064380 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36626/0 (socket says 172.21.15.26:36626) 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 -- 172.21.15.26:0/475921468 learned_addr learned my addr 172.21.15.26:0/475921468 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f967ea00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 0x7f9680114e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 -- 172.21.15.26:0/475921468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 msgr2=0x7f9680114e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 0x7f9680114e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 -- 172.21.15.26:0/475921468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668005040 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f967ea00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 0x7f9680114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.030+0000 7f9676000640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f96680047f0 tx=0x7f9668009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.031+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96680094c0 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.031+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9668032e50 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.031+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96680435f0 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.031+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968010f2a0 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.031+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968010f760 con 0x7f9680102240 2024-04-16T07:34:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.032+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800fd940 con 0x7f9680102240 2024-04-16T07:34:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.033+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f966802f070 con 0x7f9680102240 2024-04-16T07:34:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.034+0000 7f9677400640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 0x7f9664071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.034+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f96680b0960 con 0x7f9680102240 2024-04-16T07:34:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.034+0000 7f967ea00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 0x7f9664071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.035+0000 7f967ea00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 0x7f9664071020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9670004c80 tx=0x7f9670009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.039+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9668079560 con 0x7f9680102240 2024-04-16T07:34:58.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:57 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4172049658' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T07:34:58.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:57 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3001194773' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T07:34:58.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:58.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:58.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:58.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:58.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:58.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:58.190+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:58.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.284+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f9680109300 con 0x7f9680102240 2024-04-16T07:34:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.285+0000 7f9677400640 1 -- 172.21.15.26:0/475921468 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v36) v1 ==== 107+0+5007 (secure 0 0 0) 0x7f966807cc20 con 0x7f9680102240 2024-04-16T07:34:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:58.286 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:04.031949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi104.dpfdje","rank":0,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.104:6825/815493281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":815493281},{"type":"v1","addr":"172.21.15.104:6825","nonce":815493281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24287,"qdb_cluster":[24287]},"id":1}]} 2024-04-16T07:34:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2024-04-16T07:34:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 msgr2=0x7f9664071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 0x7f9664071020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9670004c80 tx=0x7f9670009040 comp rx=0 tx=0).stop 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 msgr2=0x7f96801153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f96680047f0 tx=0x7f9668009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 shutdown_connections 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f966406eb70 0x7f9664071020 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9680102240 0x7f96801153d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 --2- 172.21.15.26:0/475921468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9680064380 0x7f9680114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.288+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 >> 172.21.15.26:0/475921468 conn(0x7f96800711c0 msgr2=0x7f96800fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.289+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 shutdown_connections 2024-04-16T07:34:58.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.289+0000 7f9685a00640 1 -- 172.21.15.26:0/475921468 wait complete. 2024-04-16T07:34:58.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-16T07:34:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:58.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:58.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:58.478+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 -- 172.21.15.26:0/4182268994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fc340 msgr2=0x7f7bb0108620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 --2- 172.21.15.26:0/4182268994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fc340 0x7f7bb0108620 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0048d0 tx=0x7f7b9c009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 -- 172.21.15.26:0/4182268994 shutdown_connections 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 --2- 172.21.15.26:0/4182268994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fc340 0x7f7bb0108620 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 --2- 172.21.15.26:0/4182268994 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fba30 0x7f7bb00fbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 -- 172.21.15.26:0/4182268994 >> 172.21.15.26:0/4182268994 conn(0x7f7bb0071010 msgr2=0x7f7bb0071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:58.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.516+0000 7f7bafe00640 1 -- 172.21.15.26:0/4182268994 shutdown_connections 2024-04-16T07:34:58.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.517+0000 7f7bafe00640 1 -- 172.21.15.26:0/4182268994 wait complete. 2024-04-16T07:34:58.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 Processor -- start 2024-04-16T07:34:58.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 -- start start 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fc340 0x7f7bb0115320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0115920 con 0x7f7bb00fba30 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.518+0000 7f7bafe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb010ef70 con 0x7f7bb00fc340 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36644/0 (socket says 172.21.15.26:36644) 2024-04-16T07:34:58.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 -- 172.21.15.26:0/3115679917 learned_addr learned my addr 172.21.15.26:0/3115679917 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:58.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7bae000640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fc340 0x7f7bb0115320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 -- 172.21.15.26:0/3115679917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fc340 msgr2=0x7f7bb0115320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fc340 0x7f7bb0115320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.519+0000 7f7baea00640 1 -- 172.21.15.26:0/3115679917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c005040 con 0x7f7bb00fba30 2024-04-16T07:34:58.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7baea00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000c830 tx=0x7f7ba0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba000a850 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ba000ae80 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba0012ea0 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb010f250 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.520+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb010f740 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.521+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb00fb5d0 con 0x7f7bb00fba30 2024-04-16T07:34:58.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.522+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ba0012400 con 0x7f7bb00fba30 2024-04-16T07:34:58.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.523+0000 7f7ba6a00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 0x7f7b94071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.523+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f7ba008ebd0 con 0x7f7bb00fba30 2024-04-16T07:34:58.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.524+0000 7f7bae000640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 0x7f7b94071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.524+0000 7f7bae000640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 0x7f7b94071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00faf10 tx=0x7f7b9c009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:58.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.528+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ba00578d0 con 0x7f7bb00fba30 2024-04-16T07:34:58.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.778+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f7bb004bcc0 con 0x7f7bb00fba30 2024-04-16T07:34:58.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.779+0000 7f7ba6a00640 1 -- 172.21.15.26:0/3115679917 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v36) v1 ==== 107+0+4206 (secure 0 0 0) 0x7f7ba00126b0 con 0x7f7bb00fba30 2024-04-16T07:34:58.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:58.779 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:26.914380+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 25 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 msgr2=0x7f7b94071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 0x7f7b94071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00faf10 tx=0x7f7b9c009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 msgr2=0x7f7bb0114de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000c830 tx=0x7f7ba0004430 comp rx=0 tx=0).stop 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 shutdown_connections 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f7b9406eb70 0x7f7b94071020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb00fc340 0x7f7bb0115320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.782+0000 7f7bafe00640 1 --2- 172.21.15.26:0/3115679917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bb00fba30 0x7f7bb0114de0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.783+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 >> 172.21.15.26:0/3115679917 conn(0x7f7bb0071010 msgr2=0x7f7bb01035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.783+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 shutdown_connections 2024-04-16T07:34:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.783+0000 7f7bafe00640 1 -- 172.21.15.26:0/3115679917 wait complete. 2024-04-16T07:34:58.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4150535194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 msgr2=0x7f6a740fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4150535194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a740fc250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6a600048d0 tx=0x7f6a60009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4150535194 shutdown_connections 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4150535194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a740fc250 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4150535194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 0x7f6a741025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4150535194 >> 172.21.15.26:0/4150535194 conn(0x7f6a740f7e10 msgr2=0x7f6a740fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4150535194 shutdown_connections 2024-04-16T07:34:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.995+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4150535194 wait complete. 2024-04-16T07:34:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.996+0000 7f6a7ae00640 1 Processor -- start 2024-04-16T07:34:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.996+0000 7f6a7ae00640 1 -- start start 2024-04-16T07:34:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a7ae00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 0x7f6a74114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a7ae00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a7ae00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741159b0 con 0x7f6a74102bc0 2024-04-16T07:34:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a7ae00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7410f000 con 0x7f6a74102220 2024-04-16T07:34:58.997 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:58 smithi026 ceph-mon[92215]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:58.997 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:58 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3043694821' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T07:34:58.997 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:58 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/475921468' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T07:34:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a73400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a73400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36650/0 (socket says 172.21.15.26:36650) 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.997+0000 7f6a73400640 1 -- 172.21.15.26:0/4269501413 learned_addr learned my addr 172.21.15.26:0/4269501413 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.998+0000 7f6a73e00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 0x7f6a74114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.998+0000 7f6a73400640 1 -- 172.21.15.26:0/4269501413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 msgr2=0x7f6a74114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.998+0000 7f6a73400640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 0x7f6a74114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.998+0000 7f6a73400640 1 -- 172.21.15.26:0/4269501413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a74102bc0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a73400640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6a600047f0 tx=0x7f6a60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a600094c0 con 0x7f6a74102bc0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a60032e50 con 0x7f6a74102bc0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a600435f0 con 0x7f6a74102bc0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7410f280 con 0x7f6a74102bc0 2024-04-16T07:34:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:58.999+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7410f740 con 0x7f6a74102bc0 2024-04-16T07:34:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.001+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a740fd990 con 0x7f6a74102bc0 2024-04-16T07:34:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.001+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a6002f070 con 0x7f6a74102bc0 2024-04-16T07:34:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.003+0000 7f6a6be00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 0x7f6a58071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.003+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f6a600b07a0 con 0x7f6a74102bc0 2024-04-16T07:34:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.003+0000 7f6a73e00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 0x7f6a58071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.004+0000 7f6a73e00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 0x7f6a58071020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6a64004550 tx=0x7f6a64009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:59.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.008+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a60079490 con 0x7f6a74102bc0 2024-04-16T07:34:59.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:58 smithi104 ceph-mon[81027]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:34:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:58 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3043694821' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T07:34:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:58 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/475921468' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T07:34:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.190+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:34:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.190+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:34:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.190+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:34:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.190+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:34:59.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.191+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:34:59.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:34:59.191+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.253+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f6a74109360 con 0x7f6a74102bc0 2024-04-16T07:34:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.254+0000 7f6a6be00640 1 -- 172.21.15.26:0/4269501413 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v36) v1 ==== 107+0+4217 (secure 0 0 0) 0x7f6a6007cb50 con 0x7f6a74102bc0 2024-04-16T07:34:59.254 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:59.254 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:26.926072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24303":{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":0,"incarnation":26,"state":"up:replay","state_seq":2,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:59.254 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 26 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.257+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 msgr2=0x7f6a58071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.257+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 0x7f6a58071020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6a64004550 tx=0x7f6a64009040 comp rx=0 tx=0).stop 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.257+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 msgr2=0x7f6a741153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.257+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6a600047f0 tx=0x7f6a60009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 shutdown_connections 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f6a5806eb70 0x7f6a58071020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6a74102bc0 0x7f6a741153b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 --2- 172.21.15.26:0/4269501413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74102220 0x7f6a74114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 >> 172.21.15.26:0/4269501413 conn(0x7f6a740f7e10 msgr2=0x7f6a740f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 shutdown_connections 2024-04-16T07:34:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.258+0000 7f6a7ae00640 1 -- 172.21.15.26:0/4269501413 wait complete. 2024-04-16T07:34:59.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 -- 172.21.15.26:0/2244920106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 msgr2=0x7fa8b4064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/2244920106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4064380 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40048d0 tx=0x7fa8a4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 -- 172.21.15.26:0/2244920106 shutdown_connections 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/2244920106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 0x7fa8b4064d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/2244920106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4064380 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 -- 172.21.15.26:0/2244920106 >> 172.21.15.26:0/2244920106 conn(0x7fa8b4071010 msgr2=0x7fa8b4071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 -- 172.21.15.26:0/2244920106 shutdown_connections 2024-04-16T07:34:59.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.474+0000 7fa8b8e00640 1 -- 172.21.15.26:0/2244920106 wait complete. 2024-04-16T07:34:59.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.475+0000 7fa8b8e00640 1 Processor -- start 2024-04-16T07:34:59.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.475+0000 7fa8b8e00640 1 -- start start 2024-04-16T07:34:59.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.475+0000 7fa8b8e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 0x7fa8b41190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b8e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b8e00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4119c30 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b8e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4113280 con 0x7fa8b40648c0 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36656/0 (socket says 172.21.15.26:36656) 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 -- 172.21.15.26:0/1470123512 learned_addr learned my addr 172.21.15.26:0/1470123512 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b3400640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 0x7fa8b41190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 -- 172.21.15.26:0/1470123512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 msgr2=0x7fa8b41190f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 0x7fa8b41190f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.476+0000 7fa8b2a00640 1 -- 172.21.15.26:0/1470123512 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4005040 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8b2a00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa89c002a50 tx=0x7fa89c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa89c007c30 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa89c00f040 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b4113560 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa89c00db50 con 0x7fa8b4100f10 2024-04-16T07:34:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.477+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b4113a80 con 0x7fa8b4100f10 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:34:59.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:34:59.477+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:34:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.478+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40fb5d0 con 0x7fa8b4100f10 2024-04-16T07:34:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.480+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa89c00dcf0 con 0x7fa8b4100f10 2024-04-16T07:34:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.482+0000 7fa8ab400640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 0x7fa898071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.482+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa89c08e290 con 0x7fa8b4100f10 2024-04-16T07:34:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.482+0000 7fa8b3400640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 0x7fa898071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.483+0000 7fa8b3400640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 0x7fa898071020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4031870 tx=0x7fa8a4009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.487+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa89c056f10 con 0x7fa8b4100f10 2024-04-16T07:34:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.731+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fa8b410b4b0 con 0x7fa8b4100f10 2024-04-16T07:34:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.732+0000 7fa8ab400640 1 -- 172.21.15.26:0/1470123512 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v36) v1 ==== 107+0+4222 (secure 0 0 0) 0x7fa89c05a5d0 con 0x7fa8b4100f10 2024-04-16T07:34:59.731 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:34:59.731 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:26.954305+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24303":{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":159,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:34:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 27 2024-04-16T07:34:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 msgr2=0x7fa898071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 0x7fa898071020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4031870 tx=0x7fa8a4009da0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 msgr2=0x7fa8b4119630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa89c002a50 tx=0x7fa89c004600 comp rx=0 tx=0).stop 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 shutdown_connections 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa89806eb70 0x7fa898071020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa8b4100f10 0x7fa8b4119630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 --2- 172.21.15.26:0/1470123512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40648c0 0x7fa8b41190f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 >> 172.21.15.26:0/1470123512 conn(0x7fa8b4071010 msgr2=0x7fa8b40f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 shutdown_connections 2024-04-16T07:34:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.735+0000 7fa8b8e00640 1 -- 172.21.15.26:0/1470123512 wait complete. 2024-04-16T07:34:59.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-16T07:34:59.939 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:59 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3115679917' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T07:34:59.939 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:34:59 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4269501413' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T07:34:59.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.959+0000 7fb96f600640 1 -- 172.21.15.26:0/3362476066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9681010b0 msgr2=0x7fb968101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.959+0000 7fb96f600640 1 --2- 172.21.15.26:0/3362476066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9681010b0 0x7fb968101480 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb958009c50 tx=0x7fb958004d10 comp rx=0 tx=0).stop 2024-04-16T07:34:59.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 -- 172.21.15.26:0/3362476066 shutdown_connections 2024-04-16T07:34:59.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 --2- 172.21.15.26:0/3362476066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb968101a50 0x7fb9680fb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 --2- 172.21.15.26:0/3362476066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9681010b0 0x7fb968101480 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 -- 172.21.15.26:0/3362476066 >> 172.21.15.26:0/3362476066 conn(0x7fb9680711c0 msgr2=0x7fb9680715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:34:59.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 -- 172.21.15.26:0/3362476066 shutdown_connections 2024-04-16T07:34:59.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.960+0000 7fb96f600640 1 -- 172.21.15.26:0/3362476066 wait complete. 2024-04-16T07:34:59.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 Processor -- start 2024-04-16T07:34:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 -- start start 2024-04-16T07:34:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9681010b0 0x7fb968114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968115a50 con 0x7fb968101a50 2024-04-16T07:34:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.961+0000 7fb96f600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968115bc0 con 0x7fb9681010b0 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36672/0 (socket says 172.21.15.26:36672) 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 -- 172.21.15.26:0/3622631346 learned_addr learned my addr 172.21.15.26:0/3622631346 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb96cc00640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9681010b0 0x7fb968114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 -- 172.21.15.26:0/3622631346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9681010b0 msgr2=0x7fb968114f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9681010b0 0x7fb968114f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:34:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 -- 172.21.15.26:0/3622631346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9580098b0 con 0x7fb968101a50 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.962+0000 7fb95fe00640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb950002a50 tx=0x7fb950004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.963+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb950007c30 con 0x7fb968101a50 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.963+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb95000f040 con 0x7fb968101a50 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.963+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95000db50 con 0x7fb968101a50 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.963+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96810f250 con 0x7fb968101a50 2024-04-16T07:34:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.963+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96810f7f0 con 0x7fb968101a50 2024-04-16T07:34:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.964+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9680fc820 con 0x7fb968101a50 2024-04-16T07:34:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.965+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb95000dcf0 con 0x7fb968101a50 2024-04-16T07:34:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.967+0000 7fb965600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 0x7fb94c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:34:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.967+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fb95008e2c0 con 0x7fb968101a50 2024-04-16T07:34:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.967+0000 7fb96cc00640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 0x7fb94c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:34:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.968+0000 7fb96cc00640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 0x7fb94c071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb958004a30 tx=0x7fb95802e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:34:59.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:34:59.971+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb950056f40 con 0x7fb968101a50 2024-04-16T07:35:00.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:59 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3115679917' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T07:35:00.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:34:59 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4269501413' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T07:35:00.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:00.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:00.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:00.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:00.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:00.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:00.191+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:00.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.211+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fb968063f30 con 0x7fb968101a50 2024-04-16T07:35:00.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.212+0000 7fb965600640 1 -- 172.21.15.26:0/3622631346 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v36) v1 ==== 107+0+4219 (secure 0 0 0) 0x7fb95005a600 con 0x7fb968101a50 2024-04-16T07:35:00.212 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:00.212 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:27.936667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24303":{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":160,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:35:00.212 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 28 2024-04-16T07:35:00.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 msgr2=0x7fb94c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 0x7fb94c071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb958004a30 tx=0x7fb95802e040 comp rx=0 tx=0).stop 2024-04-16T07:35:00.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 msgr2=0x7fb968115450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb950002a50 tx=0x7fb950004600 comp rx=0 tx=0).stop 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 shutdown_connections 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fb94c06eb70 0x7fb94c071020 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb968101a50 0x7fb968115450 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 --2- 172.21.15.26:0/3622631346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9681010b0 0x7fb968114f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 >> 172.21.15.26:0/3622631346 conn(0x7fb9680711c0 msgr2=0x7fb9680fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 shutdown_connections 2024-04-16T07:35:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.215+0000 7fb96f600640 1 -- 172.21.15.26:0/3622631346 wait complete. 2024-04-16T07:35:00.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-16T07:35:00.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 -- 172.21.15.26:0/580062861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 msgr2=0x7f0b5006fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 --2- 172.21.15.26:0/580062861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b5006fc40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0b400048d0 tx=0x7f0b40009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 -- 172.21.15.26:0/580062861 shutdown_connections 2024-04-16T07:35:00.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 --2- 172.21.15.26:0/580062861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b50070180 0x7f0b501082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 --2- 172.21.15.26:0/580062861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b5006fc40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 -- 172.21.15.26:0/580062861 >> 172.21.15.26:0/580062861 conn(0x7f0b500f7b60 msgr2=0x7f0b500f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 -- 172.21.15.26:0/580062861 shutdown_connections 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.437+0000 7f0b4f200640 1 -- 172.21.15.26:0/580062861 wait complete. 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 Processor -- start 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 -- start start 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b50070180 0x7f0b50115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b50115880 con 0x7f0b5006f870 2024-04-16T07:35:00.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.438+0000 7f0b4f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b501159f0 con 0x7f0b50070180 2024-04-16T07:35:00.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:00.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36700/0 (socket says 172.21.15.26:36700) 2024-04-16T07:35:00.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 -- 172.21.15.26:0/352921532 learned_addr learned my addr 172.21.15.26:0/352921532 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 -- 172.21.15.26:0/352921532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b50070180 msgr2=0x7f0b50115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b50070180 0x7f0b50115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 -- 172.21.15.26:0/352921532 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40005040 con 0x7f0b5006f870 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4de00640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b400047f0 tx=0x7f0b40009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b400092b0 con 0x7f0b5006f870 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b4003b400 con 0x7f0b5006f870 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.439+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5010f020 con 0x7f0b5006f870 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.440+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b400435b0 con 0x7f0b5006f870 2024-04-16T07:35:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.440+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5010f4e0 con 0x7f0b5006f870 2024-04-16T07:35:00.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.441+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b50070fb0 con 0x7f0b5006f870 2024-04-16T07:35:00.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.441+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b4002f070 con 0x7f0b5006f870 2024-04-16T07:35:00.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.442+0000 7f0b46000640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 0x7f0b34071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.442+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f0b400b07d0 con 0x7f0b5006f870 2024-04-16T07:35:00.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.443+0000 7f0b4d400640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 0x7f0b34071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:00.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.443+0000 7f0b4d400640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 0x7f0b34071020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0b50110640 tx=0x7f0b38009c50 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:00.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.447+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b40079410 con 0x7f0b5006f870 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:00.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:00.478+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.681+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f0b5010f700 con 0x7f0b5006f870 2024-04-16T07:35:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.683+0000 7f0b46000640 1 -- 172.21.15.26:0/352921532 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v36) v1 ==== 107+0+4228 (secure 0 0 0) 0x7f0b4007cad0 con 0x7f0b5006f870 2024-04-16T07:35:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:00.682 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:29.934662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24303":{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":0,"incarnation":26,"state":"up:active","state_seq":161,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24303,"qdb_cluster":[24303]},"id":1}]} 2024-04-16T07:35:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 29 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.685+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 msgr2=0x7f0b34071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.685+0000 7f0b4f200640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 0x7f0b34071020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0b50110640 tx=0x7f0b38009c50 comp rx=0 tx=0).stop 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 msgr2=0x7f0b50114d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b400047f0 tx=0x7f0b40009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 shutdown_connections 2024-04-16T07:35:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f0b3406eb70 0x7f0b34071020 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b50070180 0x7f0b50115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 --2- 172.21.15.26:0/352921532 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b5006f870 0x7f0b50114d40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 >> 172.21.15.26:0/352921532 conn(0x7f0b500f7b60 msgr2=0x7f0b500f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:00.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 shutdown_connections 2024-04-16T07:35:00.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.686+0000 7f0b4f200640 1 -- 172.21.15.26:0/352921532 wait complete. 2024-04-16T07:35:00.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 -- 172.21.15.26:0/1176529869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 msgr2=0x7fd7ec06f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 --2- 172.21.15.26:0/1176529869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec06f870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0048d0 tx=0x7fd7dc009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 -- 172.21.15.26:0/1176529869 shutdown_connections 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 --2- 172.21.15.26:0/1176529869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 0x7fd7ec070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 --2- 172.21.15.26:0/1176529869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec06f870 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 -- 172.21.15.26:0/1176529869 >> 172.21.15.26:0/1176529869 conn(0x7fd7ec0f7b90 msgr2=0x7fd7ec0f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.919+0000 7fd7f1200640 1 -- 172.21.15.26:0/1176529869 shutdown_connections 2024-04-16T07:35:00.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.920+0000 7fd7f1200640 1 -- 172.21.15.26:0/1176529869 wait complete. 2024-04-16T07:35:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.920+0000 7fd7f1200640 1 Processor -- start 2024-04-16T07:35:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.921+0000 7fd7f1200640 1 -- start start 2024-04-16T07:35:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.921+0000 7fd7f1200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 0x7fd7ec119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.921+0000 7fd7f1200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.921+0000 7fd7f1200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec119c80 con 0x7fd7ec0714a0 2024-04-16T07:35:00.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.921+0000 7fd7f1200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec1132d0 con 0x7fd7ec06fdb0 2024-04-16T07:35:00.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:00.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36718/0 (socket says 172.21.15.26:36718) 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 -- 172.21.15.26:0/3062669657 learned_addr learned my addr 172.21.15.26:0/3062669657 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eb400640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 0x7fd7ec119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 -- 172.21.15.26:0/3062669657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 msgr2=0x7fd7ec119140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 0x7fd7ec119140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 -- 172.21.15.26:0/3062669657 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc005040 con 0x7fd7ec0714a0 2024-04-16T07:35:00.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.922+0000 7fd7eaa00640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8002a50 tx=0x7fd7d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.923+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d8007c30 con 0x7fd7ec0714a0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.923+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7d800f040 con 0x7fd7ec0714a0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.923+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d800dbe0 con 0x7fd7ec0714a0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.923+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec1135b0 con 0x7fd7ec0714a0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.923+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec113a80 con 0x7fd7ec0714a0 2024-04-16T07:35:00.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.924+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ec070fb0 con 0x7fd7ec0714a0 2024-04-16T07:35:00.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.925+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd7d800dd40 con 0x7fd7ec0714a0 2024-04-16T07:35:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.926+0000 7fd7e3400640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 0x7fd7d0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.927+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fd7d808e970 con 0x7fd7ec0714a0 2024-04-16T07:35:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.927+0000 7fd7eb400640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 0x7fd7d0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.927+0000 7fd7eb400640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 0x7fd7d0071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc003080 tx=0x7fd7dc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:00.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:00.932+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd7d8057610 con 0x7fd7ec0714a0 2024-04-16T07:35:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:00 smithi026 ceph-mon[92215]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:00 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1470123512' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T07:35:01.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:00 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3622631346' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T07:35:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:00 smithi104 ceph-mon[81027]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:01.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:00 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1470123512' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T07:35:01.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:00 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3622631346' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T07:35:01.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.186+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fd7ec04bcc0 con 0x7fd7ec0714a0 2024-04-16T07:35:01.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.187+0000 7fd7e3400640 1 -- 172.21.15.26:0/3062669657 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v36) v1 ==== 107+0+5078 (secure 0 0 0) 0x7fd7d805acd0 con 0x7fd7ec0714a0 2024-04-16T07:35:01.186 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:01.187 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:29.934662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24303":{"gid":24303,"name":"cephfs.smithi104.zwhiwr","rank":0,"incarnation":26,"state":"up:active","state_seq":161,"addr":"172.21.15.104:6827/2349406764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2349406764},{"type":"v1","addr":"172.21.15.104:6827","nonce":2349406764}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24303,"qdb_cluster":[24303]},"id":1}]} 2024-04-16T07:35:01.187 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 msgr2=0x7fd7d0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 0x7fd7d0071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc003080 tx=0x7fd7dc009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 msgr2=0x7fd7ec119680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8002a50 tx=0x7fd7d8004600 comp rx=0 tx=0).stop 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 shutdown_connections 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fd7d006eb70 0x7fd7d0071020 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ec0714a0 0x7fd7ec119680 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 --2- 172.21.15.26:0/3062669657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7ec06fdb0 0x7fd7ec119140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 >> 172.21.15.26:0/3062669657 conn(0x7fd7ec0f7b90 msgr2=0x7fd7ec0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.190+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 shutdown_connections 2024-04-16T07:35:01.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fd7f1200640 1 -- 172.21.15.26:0/3062669657 wait complete. 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:01.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:01.191+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:01.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 -- 172.21.15.26:0/2247387997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec102980 msgr2=0x7fcdec0fc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 --2- 172.21.15.26:0/2247387997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec102980 0x7fcdec0fc010 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80048d0 tx=0x7fcdd8009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 -- 172.21.15.26:0/2247387997 shutdown_connections 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 --2- 172.21.15.26:0/2247387997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec102980 0x7fcdec0fc010 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 --2- 172.21.15.26:0/2247387997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdec101fe0 0x7fcdec1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 -- 172.21.15.26:0/2247387997 >> 172.21.15.26:0/2247387997 conn(0x7fcdec0f7b90 msgr2=0x7fcdec0f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 -- 172.21.15.26:0/2247387997 shutdown_connections 2024-04-16T07:35:01.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.411+0000 7fcdf2800640 1 -- 172.21.15.26:0/2247387997 wait complete. 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.412+0000 7fcdf2800640 1 Processor -- start 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.412+0000 7fcdf2800640 1 -- start start 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf2800640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf2800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdec102980 0x7fcdec1151f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf2800640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1157f0 con 0x7fcdec101fe0 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf2800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec115960 con 0x7fcdec102980 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36746/0 (socket says 172.21.15.26:36746) 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 -- 172.21.15.26:0/79034780 learned_addr learned my addr 172.21.15.26:0/79034780 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 -- 172.21.15.26:0/79034780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdec102980 msgr2=0x7fcdec1151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdec102980 0x7fcdec1151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.413+0000 7fcdf1400640 1 -- 172.21.15.26:0/79034780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8005040 con 0x7fcdec101fe0 2024-04-16T07:35:01.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcdf1400640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcde0002a50 tx=0x7fcde0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:01.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcde0007c30 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcde000f040 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcde000db50 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec10eff0 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.414+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec10f510 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.415+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec0fd750 con 0x7fcdec101fe0 2024-04-16T07:35:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.416+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcde000dcf0 con 0x7fcdec101fe0 2024-04-16T07:35:01.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.417+0000 7fcde9600640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 0x7fcdd4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.417+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fcde008e290 con 0x7fcdec101fe0 2024-04-16T07:35:01.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.418+0000 7fcdf0a00640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 0x7fcdd4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:01.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.419+0000 7fcdf0a00640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 0x7fcdd4071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80043d0 tx=0x7fcdd8009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.423+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcde0056f10 con 0x7fcdec101fe0 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:01.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:01.478+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:01.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.661+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fcdec10f780 con 0x7fcdec101fe0 2024-04-16T07:35:01.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.662+0000 7fcde9600640 1 -- 172.21.15.26:0/79034780 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v36) v1 ==== 107+0+4274 (secure 0 0 0) 0x7fcde005a5d0 con 0x7fcdec101fe0 2024-04-16T07:35:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:01.662 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:53.830480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:35:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 31 2024-04-16T07:35:01.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 msgr2=0x7fcdd4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 0x7fcdd4071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80043d0 tx=0x7fcdd8009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 msgr2=0x7fcdec114cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcde0002a50 tx=0x7fcde0004600 comp rx=0 tx=0).stop 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 shutdown_connections 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fcdd406eb70 0x7fcdd4071020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdec102980 0x7fcdec1151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 --2- 172.21.15.26:0/79034780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec101fe0 0x7fcdec114cb0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 >> 172.21.15.26:0/79034780 conn(0x7fcdec0f7b90 msgr2=0x7fcdec0711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 shutdown_connections 2024-04-16T07:35:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.665+0000 7fcdf2800640 1 -- 172.21.15.26:0/79034780 wait complete. 2024-04-16T07:35:01.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 -- 172.21.15.26:0/2184822549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104230 msgr2=0x7efdc8104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 --2- 172.21.15.26:0/2184822549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104230 0x7efdc8104600 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009c50 tx=0x7efdbc004d10 comp rx=0 tx=0).stop 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 -- 172.21.15.26:0/2184822549 shutdown_connections 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 --2- 172.21.15.26:0/2184822549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104bd0 0x7efdc80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 --2- 172.21.15.26:0/2184822549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104230 0x7efdc8104600 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 -- 172.21.15.26:0/2184822549 >> 172.21.15.26:0/2184822549 conn(0x7efdc80f9fc0 msgr2=0x7efdc80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 -- 172.21.15.26:0/2184822549 shutdown_connections 2024-04-16T07:35:01.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.888+0000 7efdcea00640 1 -- 172.21.15.26:0/2184822549 wait complete. 2024-04-16T07:35:01.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.889+0000 7efdcea00640 1 Processor -- start 2024-04-16T07:35:01.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.889+0000 7efdcea00640 1 -- start start 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdcea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104230 0x7efdc8114d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdcea00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdcea00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc8115840 con 0x7efdc8104bd0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdcea00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc810ee90 con 0x7efdc8104230 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36772/0 (socket says 172.21.15.26:36772) 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 -- 172.21.15.26:0/1009189113 learned_addr learned my addr 172.21.15.26:0/1009189113 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdcd600640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104230 0x7efdc8114d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 -- 172.21.15.26:0/1009189113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104230 msgr2=0x7efdc8114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104230 0x7efdc8114d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.890+0000 7efdccc00640 1 -- 172.21.15.26:0/1009189113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc0098b0 con 0x7efdc8104bd0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdccc00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c810 tx=0x7efdb40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb400a820 con 0x7efdc8104bd0 2024-04-16T07:35:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdb400ae50 con 0x7efdc8104bd0 2024-04-16T07:35:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc810f170 con 0x7efdc8104bd0 2024-04-16T07:35:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb40115c0 con 0x7efdc8104bd0 2024-04-16T07:35:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.891+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc810f710 con 0x7efdc8104bd0 2024-04-16T07:35:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.892+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc80ff9a0 con 0x7efdc8104bd0 2024-04-16T07:35:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.893+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efdb400a980 con 0x7efdc8104bd0 2024-04-16T07:35:01.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.895+0000 7efdc5600640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 0x7efdb0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:01.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.895+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7efdb408ef20 con 0x7efdc8104bd0 2024-04-16T07:35:01.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.895+0000 7efdcd600640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 0x7efdb0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:01.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.896+0000 7efdcd600640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 0x7efdb0071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efdbc004a30 tx=0x7efdbc02e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:01.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:01.899+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efdb4057ba0 con 0x7efdc8104bd0 2024-04-16T07:35:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:01 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/352921532' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T07:35:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:01 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3062669657' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T07:35:02.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:01 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/79034780' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T07:35:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:01 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/352921532' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T07:35:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:01 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3062669657' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T07:35:02.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:01 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/79034780' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T07:35:02.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.148+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7efdc810f9e0 con 0x7efdc8104bd0 2024-04-16T07:35:02.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.149+0000 7efdc5600640 1 -- 172.21.15.26:0/1009189113 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v36) v1 ==== 107+0+4353 (secure 0 0 0) 0x7efdb405b260 con 0x7efdc8104bd0 2024-04-16T07:35:02.148 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:02.148 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:53.842987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34404":{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:35:02.149 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 32 2024-04-16T07:35:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.151+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 msgr2=0x7efdb0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.151+0000 7efdcea00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 0x7efdb0071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efdbc004a30 tx=0x7efdbc02e040 comp rx=0 tx=0).stop 2024-04-16T07:35:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.151+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 msgr2=0x7efdc8115240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.151+0000 7efdcea00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c810 tx=0x7efdb40043c0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 shutdown_connections 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7efdb006eb70 0x7efdb0071020 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc8104bd0 0x7efdc8115240 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 --2- 172.21.15.26:0/1009189113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc8104230 0x7efdc8114d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 >> 172.21.15.26:0/1009189113 conn(0x7efdc80f9fc0 msgr2=0x7efdc80fbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 shutdown_connections 2024-04-16T07:35:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.152+0000 7efdcea00640 1 -- 172.21.15.26:0/1009189113 wait complete. 2024-04-16T07:35:02.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:02.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:02.191+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 -- 172.21.15.26:0/1614359731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102bc0 msgr2=0x7f22d80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 --2- 172.21.15.26:0/1614359731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102bc0 0x7f22d80fc250 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f22c00048d0 tx=0x7f22c0009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 -- 172.21.15.26:0/1614359731 shutdown_connections 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 --2- 172.21.15.26:0/1614359731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102bc0 0x7f22d80fc250 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 --2- 172.21.15.26:0/1614359731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102220 0x7f22d81025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 -- 172.21.15.26:0/1614359731 >> 172.21.15.26:0/1614359731 conn(0x7f22d80f7e10 msgr2=0x7f22d80fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 -- 172.21.15.26:0/1614359731 shutdown_connections 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.394+0000 7f22dec00640 1 -- 172.21.15.26:0/1614359731 wait complete. 2024-04-16T07:35:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 Processor -- start 2024-04-16T07:35:02.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 -- start start 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102bc0 0x7f22d8115450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d8115a50 con 0x7f22d8102220 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22dec00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d8115bc0 con 0x7f22d8102bc0 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.395+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36794/0 (socket says 172.21.15.26:36794) 2024-04-16T07:35:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22d7e00640 1 -- 172.21.15.26:0/268196983 learned_addr learned my addr 172.21.15.26:0/268196983 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22cfe00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102bc0 0x7f22d8115450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22d7e00640 1 -- 172.21.15.26:0/268196983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102bc0 msgr2=0x7f22d8115450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22d7e00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102bc0 0x7f22d8115450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22d7e00640 1 -- 172.21.15.26:0/268196983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0005040 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.396+0000 7f22d7e00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f22c8002a50 tx=0x7f22c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.397+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22c8007c30 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.397+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22c800f040 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.397+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d810f250 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.397+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22c800dad0 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.397+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d810f740 con 0x7f22d8102220 2024-04-16T07:35:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.398+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d80fd990 con 0x7f22d8102220 2024-04-16T07:35:02.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.399+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f22c800dd40 con 0x7f22d8102220 2024-04-16T07:35:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.400+0000 7f22d4c00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 0x7f22bc071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.401+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f22c808e2d0 con 0x7f22d8102220 2024-04-16T07:35:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.401+0000 7f22cfe00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 0x7f22bc071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.402+0000 7f22cfe00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 0x7f22bc071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f22c00048a0 tx=0x7f22c0009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.406+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f22c8056ed0 con 0x7f22d8102220 2024-04-16T07:35:02.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:02.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:02.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:02.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:02.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:02.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:02.478+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:02.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.653+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f22d8064380 con 0x7f22d8102220 2024-04-16T07:35:02.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.654+0000 7f22d4c00640 1 -- 172.21.15.26:0/268196983 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v36) v1 ==== 107+0+4357 (secure 0 0 0) 0x7f22c8061030 con 0x7f22d8102220 2024-04-16T07:35:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:02.654 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:53.866884+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34404":{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:35:02.654 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2024-04-16T07:35:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 msgr2=0x7f22bc071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 0x7f22bc071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f22c00048a0 tx=0x7f22c0009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 msgr2=0x7f22d8114f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f22c8002a50 tx=0x7f22c8004600 comp rx=0 tx=0).stop 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 shutdown_connections 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f22bc06eb70 0x7f22bc071020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d8102bc0 0x7f22d8115450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 --2- 172.21.15.26:0/268196983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22d8102220 0x7f22d8114f10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 >> 172.21.15.26:0/268196983 conn(0x7f22d80f7e10 msgr2=0x7f22d806ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:02.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 shutdown_connections 2024-04-16T07:35:02.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.657+0000 7f22dec00640 1 -- 172.21.15.26:0/268196983 wait complete. 2024-04-16T07:35:02.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-16T07:35:02.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.944+0000 7f07a3e00640 1 -- 172.21.15.26:0/1660364961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 msgr2=0x7f07a4106030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.944+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1660364961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4106030 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f07900048d0 tx=0x7f0790009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 -- 172.21.15.26:0/1660364961 shutdown_connections 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1660364961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4106030 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1660364961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 0x7f07a40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 -- 172.21.15.26:0/1660364961 >> 172.21.15.26:0/1660364961 conn(0x7f07a40f6aa0 msgr2=0x7f07a40f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 -- 172.21.15.26:0/1660364961 shutdown_connections 2024-04-16T07:35:02.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 -- 172.21.15.26:0/1660364961 wait complete. 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.945+0000 7f07a3e00640 1 Processor -- start 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a3e00640 1 -- start start 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a3e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 0x7f07a4114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a3e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a3e00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4115870 con 0x7f07a40fd960 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a3e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a410eec0 con 0x7f07a40fd050 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a2000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a2000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36810/0 (socket says 172.21.15.26:36810) 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.946+0000 7f07a2000640 1 -- 172.21.15.26:0/1635728767 learned_addr learned my addr 172.21.15.26:0/1635728767 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a2a00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 0x7f07a4114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a2000640 1 -- 172.21.15.26:0/1635728767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 msgr2=0x7f07a4114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a2000640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 0x7f07a4114d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a2000640 1 -- 172.21.15.26:0/1635728767 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790005040 con 0x7f07a40fd960 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a2000640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07900041d0 tx=0x7f0790009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0790030870 con 0x7f07a40fd960 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a410f140 con 0x7f07a40fd960 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07900092e0 con 0x7f07a40fd960 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.947+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0790043400 con 0x7f07a40fd960 2024-04-16T07:35:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.948+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a410f660 con 0x7f07a40fd960 2024-04-16T07:35:02.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.949+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a4065d60 con 0x7f07a40fd960 2024-04-16T07:35:02.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.950+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f079003b420 con 0x7f07a40fd960 2024-04-16T07:35:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.951+0000 7f079aa00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 0x7f0788071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.951+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f07900b16f0 con 0x7f07a40fd960 2024-04-16T07:35:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.951+0000 7f07a2a00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 0x7f0788071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.952+0000 7f07a2a00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 0x7f0788071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0794009fd0 tx=0x7f0794009e50 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:02.956+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f079007a4d0 con 0x7f07a40fd960 2024-04-16T07:35:03.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:02 smithi026 ceph-mon[92215]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:03.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:02 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1009189113' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T07:35:03.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:02 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/268196983' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T07:35:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:02 smithi104 ceph-mon[81027]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:02 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1009189113' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T07:35:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:02 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/268196983' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.191+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.191+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.191+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.192+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:03.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:03.192+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.205+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f07a41091f0 con 0x7f07a40fd960 2024-04-16T07:35:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.206+0000 7f079aa00640 1 -- 172.21.15.26:0/1635728767 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v36) v1 ==== 107+0+4354 (secure 0 0 0) 0x7f079007db90 con 0x7f07a40fd960 2024-04-16T07:35:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:03.206 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi026.kwjewe","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6829/2163192915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":2163192915},{"type":"v1","addr":"172.21.15.26:6829","nonce":2163192915}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44291,"name":"cephfs.smithi104.dpfdje","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6825/2443726614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2443726614},{"type":"v1","addr":"172.21.15.104:6825","nonce":2443726614}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T07:19:38.524328+0000","modified":"2024-04-16T07:30:54.856393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34404":{"gid":34404,"name":"cephfs.smithi026.kqpyuh","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.26:6827/2688223128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":2688223128},{"type":"v1","addr":"172.21.15.26:6827","nonce":2688223128}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T07:35:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 34 2024-04-16T07:35:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 msgr2=0x7f0788071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 0x7f0788071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0794009fd0 tx=0x7f0794009e50 comp rx=0 tx=0).stop 2024-04-16T07:35:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 msgr2=0x7f07a4115270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07900041d0 tx=0x7f0790009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 shutdown_connections 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f078806eb70 0x7f0788071020 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07a40fd960 0x7f07a4115270 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 --2- 172.21.15.26:0/1635728767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07a40fd050 0x7f07a4114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 >> 172.21.15.26:0/1635728767 conn(0x7f07a40f6aa0 msgr2=0x7f07a40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 shutdown_connections 2024-04-16T07:35:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.210+0000 7f07a3e00640 1 -- 172.21.15.26:0/1635728767 wait complete. 2024-04-16T07:35:03.228 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-16T07:35:03.241 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-16T07:35:03.241 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:35:03.242 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T07:35:03.297 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:35:03.298 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T07:35:03.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:35:03.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:03.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 -- 172.21.15.26:0/1559335821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58072310 msgr2=0x7f9c5810c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 --2- 172.21.15.26:0/1559335821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58072310 0x7f9c5810c9a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9c480048d0 tx=0x7f9c48009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 -- 172.21.15.26:0/1559335821 shutdown_connections 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 --2- 172.21.15.26:0/1559335821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58072310 0x7f9c5810c9a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 --2- 172.21.15.26:0/1559335821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58071a00 0x7f9c58071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 -- 172.21.15.26:0/1559335821 >> 172.21.15.26:0/1559335821 conn(0x7f9c580f9fc0 msgr2=0x7f9c580fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 -- 172.21.15.26:0/1559335821 shutdown_connections 2024-04-16T07:35:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.573+0000 7f9c5f200640 1 -- 172.21.15.26:0/1559335821 wait complete. 2024-04-16T07:35:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 Processor -- start 2024-04-16T07:35:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 -- start start 2024-04-16T07:35:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58072310 0x7f9c58119650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58119c50 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.574+0000 7f9c5f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581132a0 con 0x7f9c58072310 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36832/0 (socket says 172.21.15.26:36832) 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 -- 172.21.15.26:0/3610685253 learned_addr learned my addr 172.21.15.26:0/3610685253 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 -- 172.21.15.26:0/3610685253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58072310 msgr2=0x7f9c58119650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5d400640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58072310 0x7f9c58119650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58072310 0x7f9c58119650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.575+0000 7f9c5de00640 1 -- 172.21.15.26:0/3610685253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48005040 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c5de00640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00c830 tx=0x7f9c4c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4c00a850 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c4c00ae80 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c58113580 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4c012ea0 con 0x7f9c58071a00 2024-04-16T07:35:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.576+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c58113aa0 con 0x7f9c58071a00 2024-04-16T07:35:03.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.577+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c58073140 con 0x7f9c58071a00 2024-04-16T07:35:03.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.578+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9c4c012400 con 0x7f9c58071a00 2024-04-16T07:35:03.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.579+0000 7f9c52000640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 0x7f9c40071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:03.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.579+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f9c4c08ecc0 con 0x7f9c58071a00 2024-04-16T07:35:03.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.580+0000 7f9c5d400640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 0x7f9c40071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:03.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.580+0000 7f9c5d400640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 0x7f9c40071030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:03.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.584+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9c4c094020 con 0x7f9c58071a00 2024-04-16T07:35:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.792+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c58114780 con 0x7f9c58071a00 2024-04-16T07:35:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.793+0000 7f9c52000640 1 -- 172.21.15.26:0/3610685253 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16975 (secure 0 0 0) 0x7f9c4c057940 con 0x7f9c58071a00 2024-04-16T07:35:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:03.794 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":85,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:30:53.829972+0000","last_up_change":"2024-04-16T07:29:20.046419+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T07:19:37.442413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T07:19:38.080594+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6803","nonce":214439947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6805","nonce":214439947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6809","nonce":214439947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6807","nonce":214439947}]},"public_addr":"172.21.15.26:6803/214439947","cluster_addr":"172.21.15.26:6805/214439947","heartbeat_back_addr":"172.21.15.26:6809/214439947","heartbeat_front_addr":"172.21.15.26:6807/214439947","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6811","nonce":547286504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6813","nonce":547286504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6817","nonce":547286504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6815","nonce":547286504}]},"public_addr":"172.21.15.26:6811/547286504","cluster_addr":"172.21.15.26:6813/547286504","heartbeat_back_addr":"172.21.15.26:6817/547286504","heartbeat_front_addr":"172.21.15.26:6815/547286504","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6819","nonce":466625097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6821","nonce":466625097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6825","nonce":466625097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6823","nonce":466625097}]},"public_addr":"172.21.15.26:6819/466625097","cluster_addr":"172.21.15.26:6821/466625097","heartbeat_back_addr":"172.21.15.26:6825/466625097","heartbeat_front_addr":"172.21.15.26:6823/466625097","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6801","nonce":1178751206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6803","nonce":1178751206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6807","nonce":1178751206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6805","nonce":1178751206}]},"public_addr":"172.21.15.104:6801/1178751206","cluster_addr":"172.21.15.104:6803/1178751206","heartbeat_back_addr":"172.21.15.104:6807/1178751206","heartbeat_front_addr":"172.21.15.104:6805/1178751206","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6809","nonce":6937815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6811","nonce":6937815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6815","nonce":6937815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6813","nonce":6937815}]},"public_addr":"172.21.15.104:6809/6937815","cluster_addr":"172.21.15.104:6811/6937815","heartbeat_back_addr":"172.21.15.104:6815/6937815","heartbeat_front_addr":"172.21.15.104:6813/6937815","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6817","nonce":2796309483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6819","nonce":2796309483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6823","nonce":2796309483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6821","nonce":2796309483}]},"public_addr":"172.21.15.104:6817/2796309483","cluster_addr":"172.21.15.104:6819/2796309483","heartbeat_back_addr":"172.21.15.104:6823/2796309483","heartbeat_front_addr":"172.21.15.104:6821/2796309483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6824/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/4200135895":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2153800113":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6827/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.26:0/2992109432":"2024-04-17T07:22:24.885942+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/81762164":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3568818565":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/683618636":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/776549524":"2024-04-17T07:24:49.215054+0000","172.21.15.26:6829/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:0/3830655902":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/1883401317":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.104:0/2105197119":"2024-04-17T07:23:44.851156+0000","172.21.15.104:0/1307182004":"2024-04-17T07:23:44.851156+0000","172.21.15.104:6829/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.104:6828/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6828/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:6826/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6827/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.104:6825/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/451733185":"2024-04-17T07:22:24.885942+0000","172.21.15.104:0/3503948742":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/846866607":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/282870970":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/431791760":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6826/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.104:0/2487700588":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/846866607":"2024-04-17T07:24:49.215054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.796+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 msgr2=0x7f9c40071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.796+0000 7f9c5f200640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 0x7f9c40071030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.796+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 msgr2=0x7f9c58119110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.796+0000 7f9c5f200640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00c830 tx=0x7f9c4c004430 comp rx=0 tx=0).stop 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 shutdown_connections 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f9c4006eb80 0x7f9c40071030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c58072310 0x7f9c58119650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 --2- 172.21.15.26:0/3610685253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c58071a00 0x7f9c58119110 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 >> 172.21.15.26:0/3610685253 conn(0x7f9c580f9fc0 msgr2=0x7f9c580fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 shutdown_connections 2024-04-16T07:35:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:03.797+0000 7f9c5f200640 1 -- 172.21.15.26:0/3610685253 wait complete. 2024-04-16T07:35:03.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T07:16:23.932743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T07:19:37.442413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T07:19:38.080594+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T07:35:03.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T07:35:04.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:03 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1635728767' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T07:35:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.042+0000 7f5625000640 1 -- 172.21.15.26:0/3459711061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5620102030 msgr2=0x7f5620102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.042+0000 7f5625000640 1 --2- 172.21.15.26:0/3459711061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5620102030 0x7f5620102400 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f56100048d0 tx=0x7f5610009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 -- 172.21.15.26:0/3459711061 shutdown_connections 2024-04-16T07:35:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 --2- 172.21.15.26:0/3459711061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56201029d0 0x7f56200fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 --2- 172.21.15.26:0/3459711061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5620102030 0x7f5620102400 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 -- 172.21.15.26:0/3459711061 >> 172.21.15.26:0/3459711061 conn(0x7f56200f7c20 msgr2=0x7f56200fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 -- 172.21.15.26:0/3459711061 shutdown_connections 2024-04-16T07:35:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 -- 172.21.15.26:0/3459711061 wait complete. 2024-04-16T07:35:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.043+0000 7f5625000640 1 Processor -- start 2024-04-16T07:35:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f5625000640 1 -- start start 2024-04-16T07:35:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f5625000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5620102030 0x7f5620114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f5625000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f5625000640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56201159b0 con 0x7f56201029d0 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f5625000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562010ef70 con 0x7f5620102030 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.044+0000 7f561ea00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36856/0 (socket says 172.21.15.26:36856) 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 -- 172.21.15.26:0/4040192825 learned_addr learned my addr 172.21.15.26:0/4040192825 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561f400640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5620102030 0x7f5620114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 -- 172.21.15.26:0/4040192825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5620102030 msgr2=0x7f5620114de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5620102030 0x7f5620114de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 -- 172.21.15.26:0/4040192825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610005040 con 0x7f56201029d0 2024-04-16T07:35:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.045+0000 7f561ea00640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f560c00b790 tx=0x7f560c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.046+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f560c00c700 con 0x7f56201029d0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.046+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f560c00cd30 con 0x7f56201029d0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.046+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562010f250 con 0x7f56201029d0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.046+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f560c013370 con 0x7f56201029d0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.046+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562010f770 con 0x7f56201029d0 2024-04-16T07:35:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.047+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56200fd7a0 con 0x7f56201029d0 2024-04-16T07:35:04.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.048+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f560c012070 con 0x7f56201029d0 2024-04-16T07:35:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.049+0000 7f5617400640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 0x7f5604071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.049+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f560c08e3b0 con 0x7f56201029d0 2024-04-16T07:35:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.050+0000 7f561f400640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 0x7f5604071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.051+0000 7f561f400640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 0x7f5604071030 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5610000c00 tx=0x7f5610009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.054+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f560c056fb0 con 0x7f56201029d0 2024-04-16T07:35:04.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:03 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1635728767' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T07:35:04.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 46) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:04.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:04.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 15 ==== client_session(renewcaps seq 46) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:04.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:04.192+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 46) v1 from mds.0 2024-04-16T07:35:04.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.269+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5620109250 con 0x7f56201029d0 2024-04-16T07:35:04.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.270+0000 7f5617400640 1 -- 172.21.15.26:0/4040192825 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f560c05a670 con 0x7f56201029d0 2024-04-16T07:35:04.270 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-16T07:35:04.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.273+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 msgr2=0x7f5604071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.273+0000 7f5625000640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 0x7f5604071030 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5610000c00 tx=0x7f5610009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.273+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 msgr2=0x7f5620115320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.273+0000 7f5625000640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f560c00b790 tx=0x7f560c0091d0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 shutdown_connections 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f560406eb80 0x7f5604071030 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201029d0 0x7f5620115320 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 --2- 172.21.15.26:0/4040192825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5620102030 0x7f5620114de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 >> 172.21.15.26:0/4040192825 conn(0x7f56200f7c20 msgr2=0x7f5620071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 shutdown_connections 2024-04-16T07:35:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.274+0000 7f5625000640 1 -- 172.21.15.26:0/4040192825 wait complete. 2024-04-16T07:35:04.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:04.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:04.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:04.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 -- 172.21.15.26:0/1067297501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc102250 msgr2=0x7f25cc102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 --2- 172.21.15.26:0/1067297501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc102250 0x7f25cc102620 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0048d0 tx=0x7f25bc009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 -- 172.21.15.26:0/1067297501 shutdown_connections 2024-04-16T07:35:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 --2- 172.21.15.26:0/1067297501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc0643b0 0x7f25cc0647e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 --2- 172.21.15.26:0/1067297501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc102250 0x7f25cc102620 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.521+0000 7f25d3200640 1 -- 172.21.15.26:0/1067297501 >> 172.21.15.26:0/1067297501 conn(0x7f25cc0710c0 msgr2=0x7f25cc0714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.522+0000 7f25d3200640 1 -- 172.21.15.26:0/1067297501 shutdown_connections 2024-04-16T07:35:04.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.522+0000 7f25d3200640 1 -- 172.21.15.26:0/1067297501 wait complete. 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.522+0000 7f25d3200640 1 Processor -- start 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25d3200640 1 -- start start 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25d3200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25d3200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc102250 0x7f25cc1153e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25d3200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc1159e0 con 0x7f25cc0643b0 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25d3200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc10f030 con 0x7f25cc102250 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36876/0 (socket says 172.21.15.26:36876) 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.523+0000 7f25cbe00640 1 -- 172.21.15.26:0/1738173823 learned_addr learned my addr 172.21.15.26:0/1738173823 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25cb400640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc102250 0x7f25cc1153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25cbe00640 1 -- 172.21.15.26:0/1738173823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc102250 msgr2=0x7f25cc1153e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25cbe00640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc102250 0x7f25cc1153e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25cbe00640 1 -- 172.21.15.26:0/1738173823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc005040 con 0x7f25cc0643b0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25cbe00640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0047f0 tx=0x7f25bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25bc0094c0 con 0x7f25cc0643b0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25bc032e50 con 0x7f25cc0643b0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.524+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25bc0435f0 con 0x7f25cc0643b0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.525+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc10f2b0 con 0x7f25cc0643b0 2024-04-16T07:35:04.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.525+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc10f770 con 0x7f25cc0643b0 2024-04-16T07:35:04.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.526+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc0fd950 con 0x7f25cc0643b0 2024-04-16T07:35:04.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.527+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f25bc02f070 con 0x7f25cc0643b0 2024-04-16T07:35:04.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.528+0000 7f25c3e00640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 0x7f25b0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.529+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f25bc0b08d0 con 0x7f25cc0643b0 2024-04-16T07:35:04.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.529+0000 7f25cb400640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 0x7f25b0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.529+0000 7f25cb400640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 0x7f25b0071030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25cc110760 tx=0x7f25b8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.534+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f25bc0794d0 con 0x7f25cc0643b0 2024-04-16T07:35:04.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.743+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f25cc110510 con 0x7f25cc0643b0 2024-04-16T07:35:04.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.744+0000 7f25c3e00640 1 -- 172.21.15.26:0/1738173823 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f25bc07cb90 con 0x7f25cc0643b0 2024-04-16T07:35:04.744 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 msgr2=0x7f25b0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 0x7f25b0071030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25cc110760 tx=0x7f25b8009040 comp rx=0 tx=0).stop 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 msgr2=0x7f25cc114ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0047f0 tx=0x7f25bc009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 shutdown_connections 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f25b006eb80 0x7f25b0071030 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25cc102250 0x7f25cc1153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 --2- 172.21.15.26:0/1738173823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25cc0643b0 0x7f25cc114ea0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.747+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 >> 172.21.15.26:0/1738173823 conn(0x7f25cc0710c0 msgr2=0x7f25cc0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:04.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.748+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 shutdown_connections 2024-04-16T07:35:04.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.748+0000 7f25d3200640 1 -- 172.21.15.26:0/1738173823 wait complete. 2024-04-16T07:35:04.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 -- 172.21.15.26:0/3122657566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 msgr2=0x7f02f41047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 --2- 172.21.15.26:0/3122657566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f41047e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f02e40048d0 tx=0x7f02e4009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 -- 172.21.15.26:0/3122657566 shutdown_connections 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 --2- 172.21.15.26:0/3122657566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 0x7f02f40fe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 --2- 172.21.15.26:0/3122657566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f41047e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 -- 172.21.15.26:0/3122657566 >> 172.21.15.26:0/3122657566 conn(0x7f02f40fa020 msgr2=0x7f02f40fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:04.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 -- 172.21.15.26:0/3122657566 shutdown_connections 2024-04-16T07:35:04.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.992+0000 7f02fa600640 1 -- 172.21.15.26:0/3122657566 wait complete. 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.993+0000 7f02fa600640 1 Processor -- start 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.993+0000 7f02fa600640 1 -- start start 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02fa600640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02fa600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 0x7f02f4115470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02fa600640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4115a70 con 0x7f02f4104410 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02fa600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f410f0c0 con 0x7f02f4104db0 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02f3400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02f3400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36906/0 (socket says 172.21.15.26:36906) 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02f3400640 1 -- 172.21.15.26:0/3760195744 learned_addr learned my addr 172.21.15.26:0/3760195744 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02ebe00640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 0x7f02f4115470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02f3400640 1 -- 172.21.15.26:0/3760195744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 msgr2=0x7f02f4115470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.994+0000 7f02f3400640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 0x7f02f4115470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02f3400640 1 -- 172.21.15.26:0/3760195744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4005040 con 0x7f02f4104410 2024-04-16T07:35:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02f3400640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f02e40047f0 tx=0x7f02e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02e40092b0 con 0x7f02f4104410 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02e403b400 con 0x7f02f4104410 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f410f340 con 0x7f02f4104410 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.995+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02e40435b0 con 0x7f02f4104410 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.996+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f410f800 con 0x7f02f4104410 2024-04-16T07:35:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.997+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f40ffb80 con 0x7f02f4104410 2024-04-16T07:35:04.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.998+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f02e402f070 con 0x7f02f4104410 2024-04-16T07:35:04.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.999+0000 7f02eb400640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 0x7f02d8071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:04.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.999+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f02e40b07f0 con 0x7f02f4104410 2024-04-16T07:35:04.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:04.999+0000 7f02ebe00640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 0x7f02d8071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:04.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.000+0000 7f02ebe00640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 0x7f02d8071030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02f41107f0 tx=0x7f02dc00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:05.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.004+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f02e40793f0 con 0x7f02f4104410 2024-04-16T07:35:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:04 smithi026 ceph-mon[92215]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:05.010 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:04 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3610685253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:35:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:04 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/4040192825' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:35:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:04 smithi104 ceph-mon[81027]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:04 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3610685253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:35:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:04 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/4040192825' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:35:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:05.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:05.192+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.213+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f02f4110630 con 0x7f02f4104410 2024-04-16T07:35:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.215+0000 7f02eb400640 1 -- 172.21.15.26:0/3760195744 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f02e407cab0 con 0x7f02f4104410 2024-04-16T07:35:05.214 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:35:05.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 msgr2=0x7f02d8071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 0x7f02d8071030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02f41107f0 tx=0x7f02dc00b040 comp rx=0 tx=0).stop 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 msgr2=0x7f02f4114f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f02e40047f0 tx=0x7f02e4009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 shutdown_connections 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f02d806eb80 0x7f02d8071030 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02f4104db0 0x7f02f4115470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.217+0000 7f02fa600640 1 --2- 172.21.15.26:0/3760195744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f4104410 0x7f02f4114f30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.218+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 >> 172.21.15.26:0/3760195744 conn(0x7f02f40fa020 msgr2=0x7f02f40faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.218+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 shutdown_connections 2024-04-16T07:35:05.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.218+0000 7f02fa600640 1 -- 172.21.15.26:0/3760195744 wait complete. 2024-04-16T07:35:05.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 -- 172.21.15.26:0/35603953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8071a00 msgr2=0x7fdbf8071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 --2- 172.21.15.26:0/35603953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8071a00 0x7fdbf8071dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009c50 tx=0x7fdbe8004d10 comp rx=0 tx=0).stop 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 -- 172.21.15.26:0/35603953 shutdown_connections 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 --2- 172.21.15.26:0/35603953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8072310 0x7fdbf810cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 --2- 172.21.15.26:0/35603953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8071a00 0x7fdbf8071dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 -- 172.21.15.26:0/35603953 >> 172.21.15.26:0/35603953 conn(0x7fdbf80f9fc0 msgr2=0x7fdbf80fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 -- 172.21.15.26:0/35603953 shutdown_connections 2024-04-16T07:35:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.450+0000 7fdbfe200640 1 -- 172.21.15.26:0/35603953 wait complete. 2024-04-16T07:35:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.451+0000 7fdbfe200640 1 Processor -- start 2024-04-16T07:35:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.451+0000 7fdbfe200640 1 -- start start 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbfe200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8071a00 0x7fdbf81192f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbfe200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbfe200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8119e30 con 0x7fdbf8072310 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbfe200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8119fa0 con 0x7fdbf8071a00 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbf6a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbf6a00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36914/0 (socket says 172.21.15.26:36914) 2024-04-16T07:35:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.452+0000 7fdbf6a00640 1 -- 172.21.15.26:0/2183833091 learned_addr learned my addr 172.21.15.26:0/2183833091 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbf7400640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8071a00 0x7fdbf81192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbf6a00640 1 -- 172.21.15.26:0/2183833091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8071a00 msgr2=0x7fdbf81192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbf6a00640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8071a00 0x7fdbf81192f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbf6a00640 1 -- 172.21.15.26:0/2183833091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80098b0 con 0x7fdbf8072310 2024-04-16T07:35:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbf6a00640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4002a50 tx=0x7fdbe4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe4007c30 con 0x7fdbf8072310 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbe400f040 con 0x7fdbf8072310 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8113630 con 0x7fdbf8072310 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.453+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe400db50 con 0x7fdbf8072310 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.454+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf8113bd0 con 0x7fdbf8072310 2024-04-16T07:35:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.455+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf8073140 con 0x7fdbf8072310 2024-04-16T07:35:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.456+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdbe400dcf0 con 0x7fdbf8072310 2024-04-16T07:35:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.457+0000 7fdbef400640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 0x7fdbdc071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.457+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fdbe408e1f0 con 0x7fdbf8072310 2024-04-16T07:35:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.457+0000 7fdbf7400640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 0x7fdbdc071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.458+0000 7fdbf7400640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 0x7fdbdc071030 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8004a30 tx=0x7fdbe802e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:05.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.462+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdbe4056f10 con 0x7fdbf8072310 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:05.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:05.478+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:05.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.682+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdbf8109910 con 0x7fdbf8072310 2024-04-16T07:35:05.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.683+0000 7fdbef400640 1 -- 172.21.15.26:0/2183833091 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v85) v1 ==== 81+0+2667 (secure 0 0 0) 0x7fdbe405a5d0 con 0x7fdbf8072310 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6824/815493281 2024-04-17T07:30:26.913848+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/4200135895 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2153800113 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6827/2349406764 2024-04-17T07:30:53.829920+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2992109432 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/2140801142 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/2140801142 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/81762164 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/2299982413 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:05.683 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/2299982413 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3568818565 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/683618636 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2226537508 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/776549524 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6829/1284848687 2024-04-17T07:30:00.990917+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3830655902 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2604992831 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/1883401317 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2186390596 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/2105197119 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/1307182004 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6829/2545344580 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3190492944 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6828/2545344580 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.684 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/3603264388 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2423323548 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3554885500 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6828/1284848687 2024-04-17T07:30:00.990917+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6826/29205890 2024-04-17T07:19:56.515804+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6827/29205890 2024-04-17T07:19:56.515804+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/3603264388 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/3943853548 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6825/815493281 2024-04-17T07:30:26.913848+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/451733185 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/3503948742 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/846866607 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3018594789 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/282870970 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3424857077 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:05.685 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/3943853548 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:05.686 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/431791760 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.686 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6826/2349406764 2024-04-17T07:30:53.829920+0000 2024-04-16T07:35:05.686 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/2487700588 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:05.686 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2384521556 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:05.686 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/846866607 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:listed 45 entries 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 msgr2=0x7fdbdc071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 0x7fdbdc071030 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8004a30 tx=0x7fdbe802e040 comp rx=0 tx=0).stop 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 msgr2=0x7fdbf8119830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4002a50 tx=0x7fdbe4004600 comp rx=0 tx=0).stop 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 shutdown_connections 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fdbdc06eb80 0x7fdbdc071030 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf8072310 0x7fdbf8119830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.685+0000 7fdbfe200640 1 --2- 172.21.15.26:0/2183833091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf8071a00 0x7fdbf81192f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.686+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 >> 172.21.15.26:0/2183833091 conn(0x7fdbf80f9fc0 msgr2=0x7fdbf8106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:05.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.686+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 shutdown_connections 2024-04-16T07:35:05.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.686+0000 7fdbfe200640 1 -- 172.21.15.26:0/2183833091 wait complete. 2024-04-16T07:35:05.702 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T07:35:05.702 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T07:35:05.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:35:05.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 -- 172.21.15.26:0/1657708448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380102630 msgr2=0x7fc380102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 --2- 172.21.15.26:0/1657708448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380102630 0x7fc380102aa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0048d0 tx=0x7fc36c009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 -- 172.21.15.26:0/1657708448 shutdown_connections 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 --2- 172.21.15.26:0/1657708448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380102630 0x7fc380102aa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 --2- 172.21.15.26:0/1657708448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380107560 0x7fc380107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 -- 172.21.15.26:0/1657708448 >> 172.21.15.26:0/1657708448 conn(0x7fc380071e50 msgr2=0x7fc380074260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 -- 172.21.15.26:0/1657708448 shutdown_connections 2024-04-16T07:35:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.967+0000 7fc37f200640 1 -- 172.21.15.26:0/1657708448 wait complete. 2024-04-16T07:35:05.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 Processor -- start 2024-04-16T07:35:05.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 -- start start 2024-04-16T07:35:05.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380102630 0x7fc380119040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc380119b80 con 0x7fc380107560 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.968+0000 7fc37f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc380119cf0 con 0x7fc380102630 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36934/0 (socket says 172.21.15.26:36934) 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 -- 172.21.15.26:0/908034585 learned_addr learned my addr 172.21.15.26:0/908034585 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37de00640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380102630 0x7fc380119040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 -- 172.21.15.26:0/908034585 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380102630 msgr2=0x7fc380119040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380102630 0x7fc380119040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.969+0000 7fc37d400640 1 -- 172.21.15.26:0/908034585 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c005040 con 0x7fc380107560 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc37d400640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc36c004930 tx=0x7fc36c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36c0094c0 con 0x7fc380107560 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc380113320 con 0x7fc380107560 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc36c032e50 con 0x7fc380107560 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36c0433b0 con 0x7fc380107560 2024-04-16T07:35:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.970+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3801137e0 con 0x7fc380107560 2024-04-16T07:35:05.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.971+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc380103d20 con 0x7fc380107560 2024-04-16T07:35:05.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.972+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc36c02f070 con 0x7fc380107560 2024-04-16T07:35:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.973+0000 7fc376000640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 0x7fc364071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.973+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fc36c0b0870 con 0x7fc380107560 2024-04-16T07:35:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.974+0000 7fc37de00640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 0x7fc364071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.974+0000 7fc37de00640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 0x7fc364071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fd0 tx=0x7fc370009e70 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:05.978+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc36c079510 con 0x7fc380107560 2024-04-16T07:35:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:05 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1738173823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:35:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:05 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/3760195744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:35:06.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:05 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2183833091' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T07:35:06.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:05 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1738173823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:35:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:05 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/3760195744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:35:06.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:05 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2183833091' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T07:35:06.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.185+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38004bcc0 con 0x7fc380107560 2024-04-16T07:35:06.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.189+0000 7fc376000640 1 -- 172.21.15.26:0/908034585 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16975 (secure 0 0 0) 0x7fc36c07cbd0 con 0x7fc380107560 2024-04-16T07:35:06.189 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T07:35:06.189 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":85,"fsid":"4c4a53e6-fbc0-11ee-bc8f-c7b262605968","created":"2024-04-16T07:10:47.864765+0000","modified":"2024-04-16T07:30:53.829972+0000","last_up_change":"2024-04-16T07:29:20.046419+0000","last_in_change":"2024-04-16T07:17:39.860338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:16:23.932743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T07:19:37.442413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T07:19:38.080594+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"302d9de6-c04f-40d5-af51-575b9d1cc85b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6803","nonce":214439947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6805","nonce":214439947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6809","nonce":214439947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":214439947},{"type":"v1","addr":"172.21.15.26:6807","nonce":214439947}]},"public_addr":"172.21.15.26:6803/214439947","cluster_addr":"172.21.15.26:6805/214439947","heartbeat_back_addr":"172.21.15.26:6809/214439947","heartbeat_front_addr":"172.21.15.26:6807/214439947","state":["exists","up"]},{"osd":1,"uuid":"2ddb8857-f53a-425e-982e-7f2c90978ccc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6811","nonce":547286504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6813","nonce":547286504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6817","nonce":547286504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":547286504},{"type":"v1","addr":"172.21.15.26:6815","nonce":547286504}]},"public_addr":"172.21.15.26:6811/547286504","cluster_addr":"172.21.15.26:6813/547286504","heartbeat_back_addr":"172.21.15.26:6817/547286504","heartbeat_front_addr":"172.21.15.26:6815/547286504","state":["exists","up"]},{"osd":2,"uuid":"d9ebe37f-b2ce-4c6f-b5d4-6b69d7aca1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6819","nonce":466625097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6821","nonce":466625097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6825","nonce":466625097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":466625097},{"type":"v1","addr":"172.21.15.26:6823","nonce":466625097}]},"public_addr":"172.21.15.26:6819/466625097","cluster_addr":"172.21.15.26:6821/466625097","heartbeat_back_addr":"172.21.15.26:6825/466625097","heartbeat_front_addr":"172.21.15.26:6823/466625097","state":["exists","up"]},{"osd":3,"uuid":"e51da929-01c2-498d-9c12-e4bb3ea1e4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6801","nonce":1178751206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6803","nonce":1178751206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6807","nonce":1178751206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1178751206},{"type":"v1","addr":"172.21.15.104:6805","nonce":1178751206}]},"public_addr":"172.21.15.104:6801/1178751206","cluster_addr":"172.21.15.104:6803/1178751206","heartbeat_back_addr":"172.21.15.104:6807/1178751206","heartbeat_front_addr":"172.21.15.104:6805/1178751206","state":["exists","up"]},{"osd":4,"uuid":"0d46a520-bde2-4ef9-9420-1ebb0edbe1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6809","nonce":6937815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6811","nonce":6937815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6815","nonce":6937815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":6937815},{"type":"v1","addr":"172.21.15.104:6813","nonce":6937815}]},"public_addr":"172.21.15.104:6809/6937815","cluster_addr":"172.21.15.104:6811/6937815","heartbeat_back_addr":"172.21.15.104:6815/6937815","heartbeat_front_addr":"172.21.15.104:6813/6937815","state":["exists","up"]},{"osd":5,"uuid":"7f752a2e-4458-4cde-821d-59a8d703f253","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6817","nonce":2796309483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6819","nonce":2796309483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6823","nonce":2796309483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2796309483},{"type":"v1","addr":"172.21.15.104:6821","nonce":2796309483}]},"public_addr":"172.21.15.104:6817/2796309483","cluster_addr":"172.21.15.104:6819/2796309483","heartbeat_back_addr":"172.21.15.104:6823/2796309483","heartbeat_front_addr":"172.21.15.104:6821/2796309483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:06.135176+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:15:43.124755+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:20.399198+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:16:53.840164+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:17:28.089739+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T07:18:02.203272+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6824/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/4200135895":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2153800113":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6827/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.26:0/2992109432":"2024-04-17T07:22:24.885942+0000","172.21.15.26:6800/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/2140801142":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/81762164":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6800/2299982413":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/3568818565":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/683618636":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/2226537508":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/776549524":"2024-04-17T07:24:49.215054+0000","172.21.15.26:6829/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:0/3830655902":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/2604992831":"2024-04-17T07:13:31.214833+0000","172.21.15.104:0/1883401317":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2186390596":"2024-04-17T07:12:00.578283+0000","172.21.15.104:0/2105197119":"2024-04-17T07:23:44.851156+0000","172.21.15.104:0/1307182004":"2024-04-17T07:23:44.851156+0000","172.21.15.104:6829/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/3190492944":"2024-04-17T07:13:31.214833+0000","172.21.15.104:6828/2545344580":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6800/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/2423323548":"2024-04-17T07:12:00.578283+0000","172.21.15.26:0/3554885500":"2024-04-17T07:11:19.928318+0000","172.21.15.26:6828/1284848687":"2024-04-17T07:30:00.990917+0000","172.21.15.26:6826/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6827/29205890":"2024-04-17T07:19:56.515804+0000","172.21.15.26:6801/3603264388":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.104:6825/815493281":"2024-04-17T07:30:26.913848+0000","172.21.15.26:0/451733185":"2024-04-17T07:22:24.885942+0000","172.21.15.104:0/3503948742":"2024-04-17T07:23:44.851156+0000","172.21.15.26:6801/846866607":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3018594789":"2024-04-17T07:11:19.928318+0000","172.21.15.26:0/282870970":"2024-04-17T07:24:49.215054+0000","172.21.15.26:0/3424857077":"2024-04-17T07:13:31.214833+0000","172.21.15.26:6801/3943853548":"2024-04-17T07:22:24.885942+0000","172.21.15.26:0/431791760":"2024-04-17T07:24:49.215054+0000","172.21.15.104:6826/2349406764":"2024-04-17T07:30:53.829920+0000","172.21.15.104:0/2487700588":"2024-04-17T07:23:44.851156+0000","172.21.15.26:0/2384521556":"2024-04-17T07:12:00.578283+0000","172.21.15.26:6800/846866607":"2024-04-17T07:24:49.215054+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:35:06.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 msgr2=0x7fc364071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 0x7fc364071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fd0 tx=0x7fc370009e70 comp rx=0 tx=0).stop 2024-04-16T07:35:06.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 msgr2=0x7fc380119580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc36c004930 tx=0x7fc36c009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 shutdown_connections 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fc36406eb70 0x7fc364071020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc380107560 0x7fc380119580 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 --2- 172.21.15.26:0/908034585 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc380102630 0x7fc380119040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 >> 172.21.15.26:0/908034585 conn(0x7fc380071e50 msgr2=0x7fc380104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 shutdown_connections 2024-04-16T07:35:06.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fc37f200640 1 -- 172.21.15.26:0/908034585 wait complete. 2024-04-16T07:35:06.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:06.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:06.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:06.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:06.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:06.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:06.192+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:06.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T07:16:23.932743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T07:19:37.442413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T07:19:38.080594+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T07:35:06.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T07:35:06.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 -- 172.21.15.26:0/2603460666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 msgr2=0x7fa3780643e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 --2- 172.21.15.26:0/2603460666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa3780643e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa3680048d0 tx=0x7fa368009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 -- 172.21.15.26:0/2603460666 shutdown_connections 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 --2- 172.21.15.26:0/2603460666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 0x7fa378064d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 --2- 172.21.15.26:0/2603460666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa3780643e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 -- 172.21.15.26:0/2603460666 >> 172.21.15.26:0/2603460666 conn(0x7fa378071df0 msgr2=0x7fa3780721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 -- 172.21.15.26:0/2603460666 shutdown_connections 2024-04-16T07:35:06.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.426+0000 7fa377e00640 1 -- 172.21.15.26:0/2603460666 wait complete. 2024-04-16T07:35:06.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.427+0000 7fa377e00640 1 Processor -- start 2024-04-16T07:35:06.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.427+0000 7fa377e00640 1 -- start start 2024-04-16T07:35:06.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa377e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 0x7fa3781191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa377e00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa377e00640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378119d00 con 0x7fa378100ee0 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa377e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378113350 con 0x7fa378064920 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa376000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa376000640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36948/0 (socket says 172.21.15.26:36948) 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa376000640 1 -- 172.21.15.26:0/44981097 learned_addr learned my addr 172.21.15.26:0/44981097 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:06.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.428+0000 7fa376a00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 0x7fa3781191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa376000640 1 -- 172.21.15.26:0/44981097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 msgr2=0x7fa3781191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa376000640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 0x7fa3781191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa376000640 1 -- 172.21.15.26:0/44981097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368005040 con 0x7fa378100ee0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa376000640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa36000db00 tx=0x7fa360004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa360017070 con 0x7fa378100ee0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa36000f040 con 0x7fa378100ee0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa360003a70 con 0x7fa378100ee0 2024-04-16T07:35:06.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa378113630 con 0x7fa378100ee0 2024-04-16T07:35:06.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.429+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa378113b50 con 0x7fa378100ee0 2024-04-16T07:35:06.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.431+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3780fb590 con 0x7fa378100ee0 2024-04-16T07:35:06.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.431+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa360003bd0 con 0x7fa378100ee0 2024-04-16T07:35:06.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.432+0000 7fa36ea00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 0x7fa358071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.433+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa36008eaa0 con 0x7fa378100ee0 2024-04-16T07:35:06.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.433+0000 7fa376a00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 0x7fa358071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.433+0000 7fa376a00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 0x7fa358071030 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3680048a0 tx=0x7fa368009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:06.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.437+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa360057720 con 0x7fa378100ee0 2024-04-16T07:35:06.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:06.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:06.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:06.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:06.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:06.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:06.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:06.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.643+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa378114830 con 0x7fa378100ee0 2024-04-16T07:35:06.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.644+0000 7fa36ea00640 1 -- 172.21.15.26:0/44981097 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa36005ade0 con 0x7fa378100ee0 2024-04-16T07:35:06.645 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 msgr2=0x7fa358071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 0x7fa358071030 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3680048a0 tx=0x7fa368009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 msgr2=0x7fa378119700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa36000db00 tx=0x7fa360004e60 comp rx=0 tx=0).stop 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 shutdown_connections 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa35806eb80 0x7fa358071030 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378100ee0 0x7fa378119700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 --2- 172.21.15.26:0/44981097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa378064920 0x7fa3781191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.646+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 >> 172.21.15.26:0/44981097 conn(0x7fa378071df0 msgr2=0x7fa3780f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.647+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 shutdown_connections 2024-04-16T07:35:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.647+0000 7fa377e00640 1 -- 172.21.15.26:0/44981097 wait complete. 2024-04-16T07:35:06.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T07:35:06.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.883+0000 7f877f200640 1 -- 172.21.15.26:0/806036806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 msgr2=0x7f8780108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.883+0000 7f877f200640 1 --2- 172.21.15.26:0/806036806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780108a70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f87700048d0 tx=0x7f8770009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 -- 172.21.15.26:0/806036806 shutdown_connections 2024-04-16T07:35:06.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 --2- 172.21.15.26:0/806036806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 0x7f8780102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 --2- 172.21.15.26:0/806036806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780108a70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 -- 172.21.15.26:0/806036806 >> 172.21.15.26:0/806036806 conn(0x7f8780071e70 msgr2=0x7f8780074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:06.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 -- 172.21.15.26:0/806036806 shutdown_connections 2024-04-16T07:35:06.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 -- 172.21.15.26:0/806036806 wait complete. 2024-04-16T07:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.884+0000 7f877f200640 1 Processor -- start 2024-04-16T07:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877f200640 1 -- start start 2024-04-16T07:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 0x7f8780119130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877f200640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877f200640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780119c70 con 0x7f87801086a0 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87801132c0 con 0x7f87801026d0 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877d400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877d400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36964/0 (socket says 172.21.15.26:36964) 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.885+0000 7f877d400640 1 -- 172.21.15.26:0/1547603168 learned_addr learned my addr 172.21.15.26:0/1547603168 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877de00640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 0x7f8780119130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877d400640 1 -- 172.21.15.26:0/1547603168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 msgr2=0x7f8780119130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877d400640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 0x7f8780119130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877d400640 1 -- 172.21.15.26:0/1547603168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770005040 con 0x7f87801086a0 2024-04-16T07:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877d400640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f876c002a80 tx=0x7f876c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876c007c30 con 0x7f87801086a0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f876c00f040 con 0x7f87801086a0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87801135a0 con 0x7f87801086a0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.886+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876c00db70 con 0x7f87801086a0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.887+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8780113ac0 con 0x7f87801086a0 2024-04-16T07:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.888+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8780103dc0 con 0x7f87801086a0 2024-04-16T07:35:06.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.889+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f876c00dcd0 con 0x7f87801086a0 2024-04-16T07:35:06.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.890+0000 7f8776000640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 0x7f8764071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:06.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.890+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f876c08e3b0 con 0x7f87801086a0 2024-04-16T07:35:06.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.890+0000 7f877de00640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 0x7f8764071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:06.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.891+0000 7f877de00640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 0x7f8764071030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87700048a0 tx=0x7f8770009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:06.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:06.895+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f876c056fb0 con 0x7f87801086a0 2024-04-16T07:35:07.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:06 smithi026 ceph-mon[92215]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:07.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:06 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/908034585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:35:07.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:06 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/44981097' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:35:07.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:06 smithi104 ceph-mon[81027]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:07.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:06 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/908034585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T07:35:07.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:06 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/44981097' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T07:35:07.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.101+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f87801147a0 con 0x7f87801086a0 2024-04-16T07:35:07.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.102+0000 7f8776000640 1 -- 172.21.15.26:0/1547603168 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f876c05a670 con 0x7f87801086a0 2024-04-16T07:35:07.101 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 msgr2=0x7f8764071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 0x7f8764071030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87700048a0 tx=0x7f8770009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 msgr2=0x7f8780119670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f876c002a80 tx=0x7f876c004600 comp rx=0 tx=0).stop 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 shutdown_connections 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f876406eb80 0x7f8764071030 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87801086a0 0x7f8780119670 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 --2- 172.21.15.26:0/1547603168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87801026d0 0x7f8780119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 >> 172.21.15.26:0/1547603168 conn(0x7f8780071e70 msgr2=0x7f8780073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:07.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 shutdown_connections 2024-04-16T07:35:07.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.105+0000 7f877f200640 1 -- 172.21.15.26:0/1547603168 wait complete. 2024-04-16T07:35:07.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T07:35:07.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:07.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:07.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:07.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:07.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:07.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:07.192+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 -- 172.21.15.26:0/1376348104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141026d0 msgr2=0x7fa414102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 --2- 172.21.15.26:0/1376348104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141026d0 0x7fa414102b40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa4000048d0 tx=0x7fa400009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 -- 172.21.15.26:0/1376348104 shutdown_connections 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 --2- 172.21.15.26:0/1376348104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141026d0 0x7fa414102b40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 --2- 172.21.15.26:0/1376348104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141086a0 0x7fa414108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 -- 172.21.15.26:0/1376348104 >> 172.21.15.26:0/1376348104 conn(0x7fa414071e70 msgr2=0x7fa414074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 -- 172.21.15.26:0/1376348104 shutdown_connections 2024-04-16T07:35:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.343+0000 7fa419800640 1 -- 172.21.15.26:0/1376348104 wait complete. 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 Processor -- start 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 -- start start 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141026d0 0x7fa414119180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa414119d50 con 0x7fa4141086a0 2024-04-16T07:35:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.344+0000 7fa419800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa414113310 con 0x7fa4141026d0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36994/0 (socket says 172.21.15.26:36994) 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 -- 172.21.15.26:0/1664289530 learned_addr learned my addr 172.21.15.26:0/1664289530 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413e00640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141026d0 0x7fa414119180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 -- 172.21.15.26:0/1664289530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141026d0 msgr2=0x7fa414119180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141026d0 0x7fa414119180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.345+0000 7fa413400640 1 -- 172.21.15.26:0/1664289530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400005040 con 0x7fa4141086a0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa413400640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa4000047f0 tx=0x7fa400009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4000094c0 con 0x7fa4141086a0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa400032e50 con 0x7fa4141086a0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa414113590 con 0x7fa4141086a0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4000435f0 con 0x7fa4141086a0 2024-04-16T07:35:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.346+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa414113a50 con 0x7fa4141086a0 2024-04-16T07:35:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.347+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa414103dc0 con 0x7fa4141086a0 2024-04-16T07:35:07.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.348+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa40002f070 con 0x7fa4141086a0 2024-04-16T07:35:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.350+0000 7fa40be00640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 0x7fa3f8071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.350+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa4000b08b0 con 0x7fa4141086a0 2024-04-16T07:35:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.350+0000 7fa413e00640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 0x7fa3f8071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.351+0000 7fa413e00640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 0x7fa3f8071070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa404006fd0 tx=0x7fa404008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.355+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa4000794b0 con 0x7fa4141086a0 2024-04-16T07:35:07.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:07.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:07.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:07.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:07.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:07.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:07.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:07.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.559+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa41404bcc0 con 0x7fa4141086a0 2024-04-16T07:35:07.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.560+0000 7fa40be00640 1 -- 172.21.15.26:0/1664289530 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa40007cb70 con 0x7fa4141086a0 2024-04-16T07:35:07.559 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 msgr2=0x7fa3f8071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 0x7fa3f8071070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa404006fd0 tx=0x7fa404008040 comp rx=0 tx=0).stop 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 msgr2=0x7fa4141196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa4000047f0 tx=0x7fa400009da0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 shutdown_connections 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7fa3f806ebc0 0x7fa3f8071070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa4141086a0 0x7fa4141196c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 --2- 172.21.15.26:0/1664289530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4141026d0 0x7fa414119180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.562+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 >> 172.21.15.26:0/1664289530 conn(0x7fa414071e70 msgr2=0x7fa414073cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:07.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.563+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 shutdown_connections 2024-04-16T07:35:07.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.563+0000 7fa419800640 1 -- 172.21.15.26:0/1664289530 wait complete. 2024-04-16T07:35:07.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 -- 172.21.15.26:0/2477970228 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 msgr2=0x7f4a9c06f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2477970228 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c06f870 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009c50 tx=0x7f4a8c004d10 comp rx=0 tx=0).stop 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 -- 172.21.15.26:0/2477970228 shutdown_connections 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2477970228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 0x7f4a9c070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2477970228 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c06f870 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 -- 172.21.15.26:0/2477970228 >> 172.21.15.26:0/2477970228 conn(0x7f4a9c0f7be0 msgr2=0x7f4a9c0f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 -- 172.21.15.26:0/2477970228 shutdown_connections 2024-04-16T07:35:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.794+0000 7f4aa1400640 1 -- 172.21.15.26:0/2477970228 wait complete. 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.795+0000 7f4aa1400640 1 Processor -- start 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.795+0000 7f4aa1400640 1 -- start start 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4aa1400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 0x7f4a9c1191d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4aa1400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4aa1400640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c119d10 con 0x7f4a9c0714a0 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4aa1400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c113360 con 0x7f4a9c06fdb0 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4a9b400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4a9b400640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37006/0 (socket says 172.21.15.26:37006) 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4a9b400640 1 -- 172.21.15.26:0/2016842425 learned_addr learned my addr 172.21.15.26:0/2016842425 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.796+0000 7f4a9be00640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 0x7f4a9c1191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a9b400640 1 -- 172.21.15.26:0/2016842425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 msgr2=0x7f4a9c1191d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a9b400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 0x7f4a9c1191d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a9b400640 1 -- 172.21.15.26:0/2016842425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c0098b0 con 0x7f4a9c0714a0 2024-04-16T07:35:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a9b400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a84002a50 tx=0x7f4a84004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a84007c30 con 0x7f4a9c0714a0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a8400f040 con 0x7f4a9c0714a0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a8400db50 con 0x7f4a9c0714a0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c113640 con 0x7f4a9c0714a0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.797+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c113b60 con 0x7f4a9c0714a0 2024-04-16T07:35:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.798+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c070fb0 con 0x7f4a9c0714a0 2024-04-16T07:35:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.800+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a8400dcf0 con 0x7f4a9c0714a0 2024-04-16T07:35:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.800+0000 7f4a93e00640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 0x7f4a80071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:35:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.801+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f4a8408e290 con 0x7f4a9c0714a0 2024-04-16T07:35:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.801+0000 7f4a9be00640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 0x7f4a80071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:35:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.802+0000 7f4a9be00640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 0x7f4a80071020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c004f50 tx=0x7f4a8c02e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:35:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:07.805+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a84056f10 con 0x7f4a9c0714a0 2024-04-16T07:35:08.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:07 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1547603168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:35:08.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:07 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/1664289530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:35:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.027+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4a9c10f750 con 0x7f4a9c0714a0 2024-04-16T07:35:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.028+0000 7f4a93e00640 1 -- 172.21.15.26:0/2016842425 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v85) v1 ==== 81+0+2667 (secure 0 0 0) 0x7f4a8405a5d0 con 0x7f4a9c0714a0 2024-04-16T07:35:08.027 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6824/815493281 2024-04-17T07:30:26.913848+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/4200135895 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2153800113 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6827/2349406764 2024-04-17T07:30:53.829920+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2992109432 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/2140801142 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/2140801142 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/81762164 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/2299982413 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/2299982413 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3568818565 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/683618636 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2226537508 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/776549524 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6829/1284848687 2024-04-17T07:30:00.990917+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3830655902 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2604992831 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/1883401317 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2186390596 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/2105197119 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/1307182004 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6829/2545344580 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3190492944 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6828/2545344580 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/3603264388 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2423323548 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3554885500 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6828/1284848687 2024-04-17T07:30:00.990917+0000 2024-04-16T07:35:08.029 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6826/29205890 2024-04-17T07:19:56.515804+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6827/29205890 2024-04-17T07:19:56.515804+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/3603264388 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/3943853548 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6825/815493281 2024-04-17T07:30:26.913848+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/451733185 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/3503948742 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/846866607 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3018594789 2024-04-17T07:11:19.928318+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/282870970 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3424857077 2024-04-17T07:13:31.214833+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/3943853548 2024-04-17T07:22:24.885942+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/431791760 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:6826/2349406764 2024-04-17T07:30:53.829920+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.104:0/2487700588 2024-04-17T07:23:44.851156+0000 2024-04-16T07:35:08.030 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2384521556 2024-04-17T07:12:00.578283+0000 2024-04-16T07:35:08.031 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/846866607 2024-04-17T07:24:49.215054+0000 2024-04-16T07:35:08.031 INFO:teuthology.orchestra.run.smithi026.stderr:listed 45 entries 2024-04-16T07:35:08.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 msgr2=0x7f4a80071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:08.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 0x7f4a80071020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c004f50 tx=0x7f4a8c02e040 comp rx=0 tx=0).stop 2024-04-16T07:35:08.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 msgr2=0x7f4a9c119710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a84002a50 tx=0x7f4a84004600 comp rx=0 tx=0).stop 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 shutdown_connections 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:6800/2662957189,v1:172.21.15.26:6801/2662957189] conn(0x7f4a8006eb70 0x7f4a80071020 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a9c0714a0 0x7f4a9c119710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 --2- 172.21.15.26:0/2016842425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a9c06fdb0 0x7f4a9c1191d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 >> 172.21.15.26:0/2016842425 conn(0x7f4a9c0f7be0 msgr2=0x7f4a9c0f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 shutdown_connections 2024-04-16T07:35:08.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T07:35:08.031+0000 7f4aa1400640 1 -- 172.21.15.26:0/2016842425 wait complete. 2024-04-16T07:35:08.049 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi026.front.sepia.ceph.com... 2024-04-16T07:35:08.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T07:35:08.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-16T07:35:08.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:07 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1547603168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T07:35:08.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:07 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/1664289530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T07:35:08.131 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:35:08.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:08.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:08.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:08.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:08.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:08.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:08.193+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:08.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:08.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:09.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:08 smithi026 ceph-mon[92215]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:09.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:08 smithi026 ceph-mon[92215]: from='client.? 172.21.15.26:0/2016842425' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T07:35:09.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:08 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:35:09.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:08 smithi104 ceph-mon[81027]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:09.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:08 smithi104 ceph-mon[81027]: from='client.? 172.21.15.26:0/2016842425' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T07:35:09.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:08 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:09.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:09.193+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:09.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:09.479+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:10.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:10.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:10.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:10.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:10.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:10.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:10.193+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:10.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:10.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:10.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:10.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:10.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:10.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:10.480+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:10 smithi026 ceph-mon[92215]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:35:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:35:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:35:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:10 smithi104 ceph-mon[81027]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:35:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:35:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:35:11.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:11.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:11.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:11.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:11.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:11.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:11.194+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:11.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:11.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:11.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:11.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:11.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:11.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:11.480+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:12.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:12.194+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 46) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006860 con 0x7fe9b4014240 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:12.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 15 ==== client_session(renewcaps seq 46) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:35:12.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:12.480+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 46) v1 from mds.0 2024-04-16T07:35:13.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:12 smithi026 ceph-mon[92215]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:13.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:12 smithi104 ceph-mon[81027]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:13.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:13.194+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:13.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:13.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:13.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:13.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:13.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:13.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:13.480+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:14.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:14.195+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:14.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:14.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:14.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:14.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:14.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:14.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:14.481+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:15.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:14 smithi104 ceph-mon[81027]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:15.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.194+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:15.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.195+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:15.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.195+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:15.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.195+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:15.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.195+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:15.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:15.195+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:15.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:14 smithi026 ceph-mon[92215]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:15.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:15.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:15.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:15.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:15.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:15.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:15.481+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:16.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:16.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:16.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:16.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:16.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:16.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:16.195+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:16.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:16.481+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:17.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:16 smithi104 ceph-mon[81027]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:17.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:17.195+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:17.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:16 smithi026 ceph-mon[92215]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.481+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.482+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.482+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:17.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:17.482+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:18.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:18.196+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:18.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:18.482+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:19.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:18 smithi104 ceph-mon[81027]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:19.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.195+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:19.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.196+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:19.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.196+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:19.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.196+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:19.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.196+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:19.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:19.196+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:19.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:18 smithi026 ceph-mon[92215]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:19.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:19.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:19.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:19.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:19.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:19.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:19.482+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:20.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:20.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:20.196+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:20.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:20.482+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:21.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:20 smithi104 ceph-mon[81027]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:21.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:21.197+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:20 smithi026 ceph-mon[92215]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:21.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:21.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:21.483+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:22.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:22.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:22.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:22.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:22.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:22.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:22.197+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:22.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:22.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:22.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:22.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:22.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:22.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:22.483+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:23.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:22 smithi104 ceph-mon[81027]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:23.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.107+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c004910 con 0x7fbf70008390 2024-04-16T07:35:23.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.107+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c004910 con 0x7fbf70008390 2024-04-16T07:35:23.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:23.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:23.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:23.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:23.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:23.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:23.198+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:23.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:22 smithi026 ceph-mon[92215]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:23.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:23.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:23.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:23.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:23.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:23.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:23.483+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:24.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 47) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 16 ==== client_session(renewcaps seq 47) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:24.198+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 47) v1 from mds.0 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:24.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:24.483+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:24 smithi104 ceph-mon[81027]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:25.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:25.197+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:25.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:24 smithi026 ceph-mon[92215]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:25.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:25.484+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:26.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:26.197+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:26.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:26.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:26.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:26.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:26.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:26.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:26.484+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:26 smithi104 ceph-mon[81027]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:27.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:27.198+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:27.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:26 smithi026 ceph-mon[92215]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:27.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:27.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:27.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:27.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:27.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:27.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:27.484+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:28.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:28.198+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:28.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:28.484+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:29.065 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:28 smithi026 ceph-mon[92215]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:28 smithi104 ceph-mon[81027]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:29.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:29.198+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:29.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:29.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:29.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:29.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:29.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:29.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:29.485+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:30.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.198+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:30.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.198+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:30.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.198+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:30.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.198+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:30.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.199+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:30.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:30.199+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:30.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:30.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:30.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:30.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:30.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:30.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:30.485+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:31.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:30 smithi104 ceph-mon[81027]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:31.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:31.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:31.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:31.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:31.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:31.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:31.199+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:31.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:30 smithi026 ceph-mon[92215]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:31.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.394+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a00048e0 con 0x55fc3f5009a0 2024-04-16T07:35:31.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.396+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a00048e0 con 0x55fc3f5009a0 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:31.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:31.485+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:32.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:32.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:32.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:32.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:32.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:32.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:32.199+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:32.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:35:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:35:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:35:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 47) v5 -- 0x7fe9ac006860 con 0x7fe9b4014240 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.485+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:32.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.486+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 16 ==== client_session(renewcaps seq 47) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:35:32.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:32.486+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 47) v1 from mds.0 2024-04-16T07:35:33.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:32 smithi104 ceph-mon[81027]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:33.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:33.199+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:32 smithi026 ceph-mon[92215]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:33.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:33.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:33.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:33.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:33.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:33.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:33.486+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:34.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:34.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:34.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:34.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:34.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:34.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:34.200+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:34.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:34.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:34.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:34.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:34.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:34.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:34.486+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:35.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:35.200+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:34 smithi026 ceph-mon[92215]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:34 smithi104 ceph-mon[81027]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:35.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:35.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:35.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:35.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:35.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:35.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:35.486+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:36.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:36.200+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:36.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:36.487+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:37.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:37.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:37.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:37.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:37.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:37.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:37.200+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:36 smithi026 ceph-mon[92215]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:37.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:36 smithi104 ceph-mon[81027]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:37.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:37.487+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:38.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:38.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:38.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:38.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:38.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:38.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:38.201+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:38.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:38.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:38.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:38.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:38.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:38.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:38.487+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:39.201+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:38 smithi026 ceph-mon[92215]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:39.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:38 smithi104 ceph-mon[81027]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:39.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:39.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:39.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:39.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:39.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:39.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:39.487+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:40.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:40.201+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:40.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:40.488+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:41.178 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:40 smithi026 ceph-mon[92215]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:41.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:41.201+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:41.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:40 smithi104 ceph-mon[81027]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:41.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:41.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:41.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:41.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:41.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:41.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:41.488+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:42.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:42.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:42.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:42.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:42.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:42.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:42.202+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:42.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:42.488+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:43.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:43.202+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:43.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:42 smithi026 ceph-mon[92215]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:43.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:42 smithi104 ceph-mon[81027]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:43.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:43.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:43.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:43.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:43.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:43.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:43.488+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 48) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:44.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.202+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.203+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 17 ==== client_session(renewcaps seq 48) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:35:44.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:44.203+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 48) v1 from mds.0 2024-04-16T07:35:44.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:44.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:44.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:44.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:44.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:44.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:44.489+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:45.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.202+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:45.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.202+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:45.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.202+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:45.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.202+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:45.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.203+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:45.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:45.203+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:45.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:44 smithi026 ceph-mon[92215]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:45.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:44 smithi104 ceph-mon[81027]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:45.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:45.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:45.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:45.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:45.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:45.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:45.489+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:46.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:46.203+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:46.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:46.489+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:47.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:47.203+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:47.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:46 smithi026 ceph-mon[92215]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:47.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:46 smithi104 ceph-mon[81027]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:47.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:47.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:47.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:47.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:47.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:47.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:47.489+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:48.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:48.203+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:48.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:48.490+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:49.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:48 smithi104 ceph-mon[81027]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:49.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:49.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:49.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:49.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:49.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:49.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:49.204+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:48 smithi026 ceph-mon[92215]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:49.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:49.490+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:50.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:50.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:50.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:50.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:50.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:50.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:50.204+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:50.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:35:50.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:50.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:50.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:50.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:50.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:50.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:50.490+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:51.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:51.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:51.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:51.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:51.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:51.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:51.204+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:50 smithi026 ceph-mon[92215]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:50 smithi104 ceph-mon[81027]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:51.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:51.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:51.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:51.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:51.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:51.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:51.490+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:52.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:52.204+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:52.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:52.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:35:52.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:35:52.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 48) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 17 ==== client_session(renewcaps seq 48) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006860 con 0x7fe9b4014240 2024-04-16T07:35:52.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:52.491+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 48) v1 from mds.0 2024-04-16T07:35:53.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:53.205+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:53.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:52 smithi026 ceph-mon[92215]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:53.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:52 smithi104 ceph-mon[81027]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:53.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:53.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:53.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:53.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:53.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:53.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:53.491+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:54.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:54.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:54.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:54.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:54.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:54.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:54.205+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:54.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:54.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:54.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:54.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:54.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:54.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:54.491+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:55.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:55.205+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:54 smithi026 ceph-mon[92215]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:54 smithi104 ceph-mon[81027]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:55.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:55.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:55.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:55.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:55.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:55.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:55.491+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:56.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:56.206+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0068b0 con 0x7fe9b4014240 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:56.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:56.492+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:57.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:57.206+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:56 smithi026 ceph-mon[92215]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:57.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:56 smithi104 ceph-mon[81027]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:57.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:57.492+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:58.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:58.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:58.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:58.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:58.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:58.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:58.206+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:58.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:58.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:58.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:58.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:35:58.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:58.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:58.492+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:35:59.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:35:59.207+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:35:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:35:58 smithi026 ceph-mon[92215]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:59.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:35:58 smithi104 ceph-mon[81027]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:35:59.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:35:59.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:35:59.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:35:59.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:35:59.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:35:59.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:35:59.492+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:00.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:00.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:00.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:00.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:00.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:00.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:00.206+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:00.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:00.492+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:01.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:01.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:01.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:01.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:01.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:01.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:01.207+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:00 smithi026 ceph-mon[92215]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:01.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:00 smithi104 ceph-mon[81027]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:01.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:01.493+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:02.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:02.207+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:02.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:02.493+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:03.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:03.208+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:02 smithi026 ceph-mon[92215]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:03.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:02 smithi104 ceph-mon[81027]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:03.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:03.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:03.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:03.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:03.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:03.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:03.494+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 49) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:04.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:04.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:04.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 18 ==== client_session(renewcaps seq 49) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:36:04.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:04.208+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 49) v1 from mds.0 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:04.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:04.494+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:05.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:05.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:05.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:05.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:05.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:05.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:05.208+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:04 smithi026 ceph-mon[92215]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:05.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:04 smithi104 ceph-mon[81027]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:05.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:05.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:05.494+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:06.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:06.208+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:06.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:06.494+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:07.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:07.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:07.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:07.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:07.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:07.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:07.209+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:06 smithi026 ceph-mon[92215]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:07.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:06 smithi104 ceph-mon[81027]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:07.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:07.494+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:08.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:08.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:08.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:08.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:08.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:08.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:08.209+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:08.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:08.495+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:09.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:09.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:09.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:09.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:09.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:09.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:09.209+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:08 smithi026 ceph-mon[92215]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:09.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:08 smithi104 ceph-mon[81027]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:09.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:09.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:09.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:09.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:09.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:09.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:09.495+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:10.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:10.210+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:10.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:10.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:10.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:10.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:10.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:10.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:10.495+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:11.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:11.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:11.209+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:11.222 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:10 smithi026 ceph-mon[92215]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:11.222 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:10 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:36:11.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:10 smithi104 ceph-mon[81027]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:11.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:10 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:36:11.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:11.495+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:12.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:12.210+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:12.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:11 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:36:12.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:11 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:36:12.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:11 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:36:12.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:11 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:36:12.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:11 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:36:12.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:11 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 49) v5 -- 0x7fe9ac006be0 con 0x7fe9b4014240 2024-04-16T07:36:12.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 18 ==== client_session(renewcaps seq 49) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:36:12.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:12.496+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 49) v1 from mds.0 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:13.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:13.211+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:13.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:12 smithi026 ceph-mon[92215]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:13.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:12 smithi104 ceph-mon[81027]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:13.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:13.496+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:14.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:14.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:14.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:14.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:14.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:14.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:14.210+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:14.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:14.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:14.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:14.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:14.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:14.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:14.496+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:15.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:15.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:15.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:15.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:15.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:15.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:15.211+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:15.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:14 smithi026 ceph-mon[92215]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:15.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:14 smithi104 ceph-mon[81027]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:15.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:15.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:15.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:15.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:15.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:15.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:15.497+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:16.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:16.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:16.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:16.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:16.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:16.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:16.211+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:16.497+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:17.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:17.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:17.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:17.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:17.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:17.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:17.212+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:17.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:16 smithi026 ceph-mon[92215]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:17.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:16 smithi104 ceph-mon[81027]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:17.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:17.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:17.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:17.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:17.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:17.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:17.497+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:18.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:18.211+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:18.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:18.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:18.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:18.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:18.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:18.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:18.497+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:19.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.211+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:19.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.212+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:19.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.212+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:19.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.212+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:19.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.212+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:19.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:19.212+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:19.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:18 smithi026 ceph-mon[92215]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:19.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:18 smithi104 ceph-mon[81027]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:19.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:19.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:19.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:19.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:19.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:19.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:19.498+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:20.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:20.212+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:20.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:20.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:20.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:20.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:20.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:20.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:20.498+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:21.213+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:21 smithi026 ceph-mon[92215]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:21.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:21 smithi104 ceph-mon[81027]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:21.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:21.498+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:22.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:22.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:22.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:22.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:22.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:22.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:22.213+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:22.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:22.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:22.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:22.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:22.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:22.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:22.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:23.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:23.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:23.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:23.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:23.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:23.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:23.213+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:23.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:23 smithi104 ceph-mon[81027]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:23.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:23.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:23.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:23.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:23 smithi026 ceph-mon[92215]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 50) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:24.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:24.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.213+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:24.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.214+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 19 ==== client_session(renewcaps seq 50) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:36:24.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:24.214+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 50) v1 from mds.0 2024-04-16T07:36:24.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:24.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:24.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:24.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:24.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:24.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:24.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:25.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:25.214+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:25.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:25 smithi104 ceph-mon[81027]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:25.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:25.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:25.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:25 smithi026 ceph-mon[92215]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:26.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:26.214+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:26.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:26.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:26.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:26.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:26.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:26.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:26.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:27.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:27.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:27.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:27.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:27.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:27.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:27.215+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:27.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:27 smithi104 ceph-mon[81027]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:27.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:27.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:27.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:27.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:27.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:27.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:27.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:27.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:27 smithi026 ceph-mon[92215]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:28.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:28.214+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:28.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:28.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:29.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:29.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:29.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:29.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:29.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:29.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:29.214+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:29.316 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:29 smithi026 ceph-mon[92215]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:29 smithi104 ceph-mon[81027]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:29.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:29.498+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:30.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.214+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:30.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.214+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.214+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.214+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.215+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:30.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:30.215+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:30.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:30.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:30.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:30.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:30.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:30.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:30.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:31.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:31.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:31.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:31.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:31.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:31.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:31.215+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:31 smithi104 ceph-mon[81027]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:31.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:31.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:31.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:31.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:31.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:31.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:31.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:31 smithi026 ceph-mon[92215]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:32.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:32.215+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 50) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:32.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:32.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.499+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:32.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.500+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 19 ==== client_session(renewcaps seq 50) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:36:32.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:32.500+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 50) v1 from mds.0 2024-04-16T07:36:33.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:33.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:33.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:33.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:33.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:33.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:33.215+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:33.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:33 smithi104 ceph-mon[81027]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:33.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:33.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:33.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:33.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:33.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:33.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:33.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:33.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:33 smithi026 ceph-mon[92215]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:34.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:34.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:34.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:34.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:34.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:34.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:34.216+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:34.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:34.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:34.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:34.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:34.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:34.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:34.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:35.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:35.216+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:35 smithi104 ceph-mon[81027]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:35.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:35.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:35.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:35.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:35.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:35.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:35.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:35.500+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:35 smithi026 ceph-mon[92215]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:36.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:36.216+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:36.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:36.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:36.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:36.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:36.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:36.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:36.501+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:37.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:37.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:37.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:37.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:37.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:37.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:37.216+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:37.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:37 smithi104 ceph-mon[81027]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:37.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:37.501+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:37.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:37 smithi026 ceph-mon[92215]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:38.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:38.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:38.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:38.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:38.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:38.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:38.217+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:38.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:38.501+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:39.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:39.217+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:39.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:39 smithi104 ceph-mon[81027]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:39.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:39.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:39.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:39.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:39.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:39.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:39.502+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:39.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:39 smithi026 ceph-mon[92215]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:40.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:40.217+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:40.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:40.502+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:41.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:41.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:41.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:41.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:41.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:41.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:41.218+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:41.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:41 smithi104 ceph-mon[81027]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.502+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.502+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.502+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.503+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:41.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:41.503+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:41.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:41 smithi026 ceph-mon[92215]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:42.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:42.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:42.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:42.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:42.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:42.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:42.218+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:42.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:42.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:42.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:42.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:42.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:42.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:42.503+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:43.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:43.218+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:43.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:43 smithi104 ceph-mon[81027]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:43.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:43.503+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:43.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:43 smithi026 ceph-mon[92215]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:44.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:44.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:36:44.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:36:44.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 51) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.218+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.219+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 20 ==== client_session(renewcaps seq 51) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:36:44.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:44.219+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 51) v1 from mds.0 2024-04-16T07:36:44.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:44.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:44.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:44.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:44.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:44.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:44.503+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:45.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:45.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:45.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:45.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:45.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:45.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:45.219+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:45.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:45 smithi104 ceph-mon[81027]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:45.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.503+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:45.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:45.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:45.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.504+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:45.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.504+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:45.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:45.504+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:45.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:45 smithi026 ceph-mon[92215]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:46.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:46.219+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:46.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:46.504+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:47.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:47.219+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:47.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:47.504+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:47.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:47 smithi026 ceph-mon[92215]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:47.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:47 smithi104 ceph-mon[81027]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:48.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:48.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:48.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:48.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:48.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:48.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:48.219+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:48.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:48.504+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:49.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:49.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:49.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:49.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:49.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:49.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:49.220+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:49.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:49 smithi104 ceph-mon[81027]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:49.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:49.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:49.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:49.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:49.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:49.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:49.505+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:49 smithi026 ceph-mon[92215]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:50.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:50.220+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:50.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:50.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:50.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:50.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:50.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:50.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:50.505+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:50.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:51.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:51.220+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:51.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:51.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:51.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:51.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:36:51.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:51.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:51.505+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:51 smithi026 ceph-mon[92215]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:51.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:51 smithi104 ceph-mon[81027]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:52.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.220+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:52.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.220+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:52.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.220+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:52.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.220+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:52.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.221+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:52.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:52.221+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 51) v5 -- 0x7fe9ac006be0 con 0x7fe9b4014240 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.505+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:52.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.506+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 20 ==== client_session(renewcaps seq 51) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:36:52.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:52.506+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 51) v1 from mds.0 2024-04-16T07:36:53.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:53.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:53.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:53.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:53.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:53.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:53.221+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:53.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:53.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:53.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:53.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:53.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:53.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:53.506+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:53 smithi026 ceph-mon[92215]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:53 smithi104 ceph-mon[81027]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:54.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:54.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:54.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:54.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:54.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:54.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:54.221+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:54.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:54.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:54.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:54.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:54.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:54.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:54.506+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:55.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:55.221+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:55.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:55.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:55.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:55.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:55.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:55.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:55.506+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:55.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:55 smithi026 ceph-mon[92215]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:55.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:55 smithi104 ceph-mon[81027]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:56.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:56.222+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:56.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:56.506+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:57.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:57.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:57.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:57.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:57.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:57.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:57.222+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:57.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:57.507+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:57.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:57 smithi026 ceph-mon[92215]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:57 smithi104 ceph-mon[81027]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:58.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:58.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:58.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:58.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:58.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:58.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:58.222+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:58.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:58.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:58.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:58.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:58.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:58.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:58.507+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:59.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:36:59.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:36:59.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:36:59.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:36:59.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:36:59.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:36:59.223+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:59.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:36:59.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:36:59.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:36:59.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:36:59.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:36:59.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:36:59.507+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:36:59.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:36:59 smithi026 ceph-mon[92215]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:36:59.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:36:59 smithi104 ceph-mon[81027]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:00.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:00.223+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:00.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:00.507+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:01.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:01.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:01.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:01.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:01.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:01.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:01.223+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:01.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:01.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:01.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:01.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:01.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:01.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:01.508+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:01.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:01 smithi026 ceph-mon[92215]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:01.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:01 smithi104 ceph-mon[81027]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:02.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:02.224+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:02.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:02.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:02.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:02.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:02.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:02.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:02.508+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:03.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:03.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:03.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:03.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:03.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:03.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:03.223+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:03.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:03.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:03.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:03.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:03.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:03.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:03.508+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:03.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:03 smithi026 ceph-mon[92215]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:03.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:03 smithi104 ceph-mon[81027]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:04.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 52) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:04.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.224+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:04.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.225+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 21 ==== client_session(renewcaps seq 52) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:37:04.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:04.225+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 52) v1 from mds.0 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:04.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:04.509+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:05.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:05.224+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:05.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:05.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:05.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:05.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:05.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:05.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:05.509+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:05 smithi026 ceph-mon[92215]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:05.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:05 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:05.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:05 smithi104 ceph-mon[81027]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:05.584 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:05 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:06.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:06.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:06.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:06.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:06.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:06.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:06.224+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:06.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:06.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:06.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:06.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:06.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:06.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:06.509+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:06.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:06 smithi026 ceph-mon[92215]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:06.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:06 smithi104 ceph-mon[81027]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:07.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:07.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:07.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:07.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:07.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:07.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:07.225+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:07.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:07.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:07.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:07.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:07.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:07.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:07.509+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:08.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:08.225+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:08.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:08.510+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:08 smithi026 ceph-mon[92215]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:09.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:08 smithi104 ceph-mon[81027]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:09.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:09.225+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:09.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:09.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:09.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:09.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:09.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:09.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:09.510+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:10.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:10.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:10.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:10.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:10.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:10.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:10.226+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:10.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:10.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:10.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:10.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:10.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:10.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:10.510+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:11.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:10 smithi026 ceph-mon[92215]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:10 smithi104 ceph-mon[81027]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:11.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:11.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:11.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:11.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:11.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:11.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:11.226+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:11.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:11.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:11.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:11.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:11.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:11.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:11.510+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:12.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:12.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:12.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:12.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:12.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:12.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:12.226+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 52) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:12.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:12.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:12.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:12.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.510+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:12.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.511+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 21 ==== client_session(renewcaps seq 52) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006be0 con 0x7fe9b4014240 2024-04-16T07:37:12.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:12.511+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 52) v1 from mds.0 2024-04-16T07:37:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:12 smithi026 ceph-mon[92215]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:12.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:12 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:37:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:12 smithi104 ceph-mon[81027]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:12 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:13.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:13.226+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:13.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:13.511+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:14.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:13 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:37:14.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:13 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:37:14.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:13 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:37:14.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:13 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:37:14.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:13 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:37:14.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:13 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:37:14.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:14.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:14.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:14.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:14.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:14.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:14.227+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:14.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:14.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:14.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:14.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:14.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:14.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:14.511+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:15.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:14 smithi026 ceph-mon[92215]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:15.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:14 smithi104 ceph-mon[81027]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:15.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:15.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:15.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:15.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:15.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:15.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:15.227+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:15.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:15.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:15.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:15.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:15.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:15.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:15.512+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:16.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:16.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:16.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:16.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:16.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:16.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:16.227+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:16.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:16.512+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:17.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:16 smithi026 ceph-mon[92215]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:17.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:16 smithi104 ceph-mon[81027]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:17.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:17.228+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:17.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:17.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:17.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:17.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:17.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:17.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:17.512+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:18.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:18.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:18.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:18.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:18.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:18.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:18.228+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:18.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:18.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:18.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:18.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:18.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:18.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:18.512+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:19.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:18 smithi026 ceph-mon[92215]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:19.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:18 smithi104 ceph-mon[81027]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:19.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:19.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:19.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:19.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:19.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:19.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:19.228+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:19.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:19.513+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:19.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:20.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:20.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:20.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:20.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:20.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:20.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:20.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:20.229+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.512+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.512+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.512+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.513+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:20.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:20.513+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:21.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:20 smithi026 ceph-mon[92215]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:21.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:20 smithi104 ceph-mon[81027]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:21.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:21.229+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:21.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:21.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:21.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:21.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:21.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:21.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:21.513+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:22.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:22.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:22.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:22.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:22.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:22.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:22.229+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:22.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:22.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:22.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:22.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:22.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:22.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:22.513+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:22.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:22 smithi026 ceph-mon[92215]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:23.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:22 smithi104 ceph-mon[81027]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:23.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:23.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:23.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:23.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:23.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:23.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:23.229+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:23.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:23.514+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 53) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:24.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.229+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:24.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.230+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 22 ==== client_session(renewcaps seq 53) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:37:24.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:24.230+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 53) v1 from mds.0 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:24.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:24.514+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:25.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:24 smithi026 ceph-mon[92215]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:25.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:24 smithi104 ceph-mon[81027]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:25.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:25.230+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:25.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:25.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:25.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:25.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:25.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:25.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:25.514+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.229+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.229+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.229+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.229+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.230+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:26.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:26.230+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:26.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:26.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:26.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:26.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:26.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:26.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:26.515+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:27.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:26 smithi026 ceph-mon[92215]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:27.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:26 smithi104 ceph-mon[81027]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:27.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:27.230+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:27.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:27.515+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:28.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:28.231+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:28.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:28.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:28.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:28.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:28.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:28.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:28.515+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:29.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:28 smithi026 ceph-mon[92215]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:29.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:28 smithi104 ceph-mon[81027]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:29.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:29.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:29.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:29.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:29.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:29.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:29.231+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:29.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:29.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:29.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:29.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:29.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:29.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:29.515+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:30.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:30.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:30.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:30.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:30.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:30.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:30.231+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:30.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:30.516+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:31.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:30 smithi026 ceph-mon[92215]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:31.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:30 smithi104 ceph-mon[81027]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:31.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:31.232+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:31.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:31.516+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:32.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:32.231+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:32.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 53) v5 -- 0x7fe9ac006be0 con 0x7fe9b4014240 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:32.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.516+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:32.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.517+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 22 ==== client_session(renewcaps seq 53) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:32.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:32.517+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 53) v1 from mds.0 2024-04-16T07:37:32.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:32 smithi026 ceph-mon[92215]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:33.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:32 smithi104 ceph-mon[81027]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:33.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:33.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:33.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:33.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:33.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:33.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:33.231+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:33.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:33.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:33.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:33.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:33.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:33.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:33.516+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:34.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.231+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:34.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.231+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:34.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.231+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:34.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.231+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:34.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.232+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:34.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:34.232+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:34.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:34.517+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:35.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:34 smithi026 ceph-mon[92215]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:35.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:34 smithi104 ceph-mon[81027]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:35.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:35.232+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:35.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:35.517+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:36.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:36.232+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:36.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:36.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:36.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:36.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:36.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:36.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:36.517+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:37.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:36 smithi026 ceph-mon[92215]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:37.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:36 smithi104 ceph-mon[81027]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:37.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:37.232+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:37.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:37.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:37.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:37.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:37.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:37.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:37.518+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:38.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:38.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:38.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:38.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:38.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:38.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:38.233+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:38.518+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:39.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:38 smithi026 ceph-mon[92215]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:39.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:38 smithi104 ceph-mon[81027]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:39.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:39.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:39.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:39.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:39.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:39.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:39.233+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:39.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:39.518+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:40.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:40.233+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:40.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:40.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:40.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:40.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:40.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:40.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:40.518+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:41.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:40 smithi026 ceph-mon[92215]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:41.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:40 smithi104 ceph-mon[81027]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:41.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:41.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:41.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:41.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:41.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:41.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:41.233+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:41.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:41.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:41.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:41.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:41.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:41.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:41.519+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:42.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:42.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:42.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:42.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:42.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:42.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:42.234+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:42.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:42.519+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:43.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:42 smithi026 ceph-mon[92215]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:43.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:42 smithi104 ceph-mon[81027]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:43.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:43.234+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:43.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:43.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:43.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:43.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:43.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:43.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:43.519+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 54) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:44.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.234+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:44.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.235+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 23 ==== client_session(renewcaps seq 54) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:37:44.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:44.235+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 54) v1 from mds.0 2024-04-16T07:37:44.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:44.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:44.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:44.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:44.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:44.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:44.519+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:45.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:44 smithi026 ceph-mon[92215]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:45.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:44 smithi104 ceph-mon[81027]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:45.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:45.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:45.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:45.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:45.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:45.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:45.235+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:45.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:45.520+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:46.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:46.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:46.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:46.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:46.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:46.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:46.235+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:46.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:46.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:46.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:46.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:46.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:46.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:46.520+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:47.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:46 smithi026 ceph-mon[92215]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:47.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:46 smithi104 ceph-mon[81027]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:47.235+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:47.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:47.520+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:48.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:48.235+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:48.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:48.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:48.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:48.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:49.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:48 smithi026 ceph-mon[92215]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:49.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:48 smithi104 ceph-mon[81027]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:49.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:49.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:49.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:49.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:49.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:49.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:49.236+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:49.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:49.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:49.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:50.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:37:50.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:50.236+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:50.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:50.520+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:51.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:50 smithi026 ceph-mon[92215]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:51.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:50 smithi104 ceph-mon[81027]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:51.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:51.236+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:51.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:51.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:51.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:51.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:51.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:51.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:51.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:52.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:52.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:52.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:52.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:52.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:52.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:52.236+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:52.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 54) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006300 con 0x7fe9b4014240 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:52.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:52.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 23 ==== client_session(renewcaps seq 54) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006be0 con 0x7fe9b4014240 2024-04-16T07:37:52.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:52.521+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 54) v1 from mds.0 2024-04-16T07:37:53.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:52 smithi026 ceph-mon[92215]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:53.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:52 smithi104 ceph-mon[81027]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:53.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:53.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:53.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:53.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:53.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:53.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:53.237+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:53.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:53.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:54.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:54.237+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:54.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:54.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:55.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:54 smithi026 ceph-mon[92215]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:55.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:54 smithi104 ceph-mon[81027]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:55.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:55.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:55.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:55.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:55.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:55.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:55.237+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:55.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:55.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:55.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:55.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:55.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:55.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:55.521+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:56.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:56.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:56.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:56.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:56.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:56.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:56.238+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:56.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:56.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:56.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:56.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:56.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:56.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:56.522+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:57.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:56 smithi104 ceph-mon[81027]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:57.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:57.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:57.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:57.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:57.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:57.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:57.238+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:56 smithi026 ceph-mon[92215]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:57.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:57.522+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:58.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:58.238+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:58.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:58.522+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:59.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:37:58 smithi104 ceph-mon[81027]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:59.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:37:59.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:37:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:37:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:37:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:37:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:37:59.239+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:37:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:37:58 smithi026 ceph-mon[92215]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:37:59.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:37:59.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:37:59.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:37:59.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:37:59.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:37:59.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:37:59.523+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:00.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:00.239+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:00.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:00.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:00.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:00.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:00.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:00.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:00.523+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:01.063 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:00 smithi026 ceph-mon[92215]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:01.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:00 smithi104 ceph-mon[81027]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:01.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:01.239+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:01.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:01.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:01.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:01.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:01.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:01.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:01.523+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:02.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.239+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:02.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.239+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:02.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.239+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:02.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.239+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:02.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.240+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:02.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:02.240+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:02.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:02.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:02.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:02.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:02.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:02.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:02.523+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:03.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:02 smithi104 ceph-mon[81027]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:03.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:03.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:03.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:03.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:03.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:03.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:03.241+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:02 smithi026 ceph-mon[92215]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:03.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:03.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:03.524+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:04.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 55) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:04.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.241+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:04.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.242+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 24 ==== client_session(renewcaps seq 55) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:04.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:04.242+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 55) v1 from mds.0 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:04.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:04.524+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:04 smithi104 ceph-mon[81027]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:05.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:05.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:05.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:05.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:05.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:05.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:05.241+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:04 smithi026 ceph-mon[92215]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:05.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:05.524+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:06.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:06.242+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:06.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:06.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:06.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:06.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:06.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:06.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:06.525+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:07.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:06 smithi104 ceph-mon[81027]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:07.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:07.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:07.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:07.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:07.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:07.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:07.242+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:06 smithi026 ceph-mon[92215]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:07.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:07.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:07.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:07.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:07.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:07.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:07.525+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:08.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:08.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:08.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:08.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:08.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:08.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:08.242+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:08.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:08.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:08.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:08.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:08.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:08.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:08.525+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:09.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:08 smithi104 ceph-mon[81027]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:09.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:09.242+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:08 smithi026 ceph-mon[92215]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:09.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:09.525+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:10.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:10.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:10.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:10.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:10.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:10.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:10.242+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:10.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:10.526+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:10 smithi104 ceph-mon[81027]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:11.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:11.243+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:11.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:10 smithi026 ceph-mon[92215]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:11.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:11.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:11.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:11.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:11.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:11.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:11.526+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:12.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:12.244+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 55) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006c30 con 0x7fe9b4014240 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:12.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.526+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:12.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.527+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 24 ==== client_session(renewcaps seq 55) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:12.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:12.527+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 55) v1 from mds.0 2024-04-16T07:38:13.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:12 smithi104 ceph-mon[81027]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:13.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:13.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:13.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:13.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:13.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:13.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:13.244+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:13.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:12 smithi026 ceph-mon[92215]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:13.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:13.526+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:14.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:14.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:14.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:14.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:14.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:14.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:14.244+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:14.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:14.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:14.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:14.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:14.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:14.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:14.527+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:15.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:14 smithi104 ceph-mon[81027]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:15.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:14 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:15.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:15.244+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:15.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:14 smithi026 ceph-mon[92215]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:15.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:14 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:38:15.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:15.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:15.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:15.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:15.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:15.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:15.527+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:15 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:38:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:15 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:38:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:15 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:16.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:16.245+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:15 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:38:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:15 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:38:16.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:15 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:38:16.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:16.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:16.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:16.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:16.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:16.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:16.527+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:17.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:16 smithi104 ceph-mon[81027]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:17.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:17.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:17.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:17.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:17.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:17.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:17.245+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:17.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:16 smithi026 ceph-mon[92215]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:17.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:17.527+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:18.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:18.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:18.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:18.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:18.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:18.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:18.245+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:18.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:18.528+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:19.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:18 smithi104 ceph-mon[81027]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:19.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:19.245+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:19.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:18 smithi026 ceph-mon[92215]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:19.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:19.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:19.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:19.528+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:20.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:20.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:20.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:20.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:20.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:20.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:20.246+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:20.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:19 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:19 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:20.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:20.528+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:21.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:21.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:21.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:21.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:21.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:21.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:21.246+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:21.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:20 smithi026 ceph-mon[92215]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:21.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:20 smithi104 ceph-mon[81027]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:21.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:21.528+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:22.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:22.247+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:22.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:22.529+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:23.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.135+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c004af0 con 0x7fbf70008390 2024-04-16T07:38:23.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.136+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.0 v2:172.21.15.26:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fbf5c004af0 con 0x7fbf70008390 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:23.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:23.247+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:23.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:22 smithi026 ceph-mon[92215]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:23.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:22 smithi104 ceph-mon[81027]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:23.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:23.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:23.529+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 56) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:24.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 25 ==== client_session(renewcaps seq 56) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:24.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:24.248+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 56) v1 from mds.0 2024-04-16T07:38:24.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:24.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:24.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:24.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:24.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:24.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:24.529+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:25.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:25.247+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:25.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:24 smithi026 ceph-mon[92215]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:25.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:24 smithi104 ceph-mon[81027]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:25.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:25.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:25.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:25.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:25.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:25.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:25.529+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:26.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:26.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:26.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:26.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:26.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:26.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:26.247+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:26.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:26.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:26.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:26.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:26.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:26.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:26.530+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:27.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:27.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:27.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:27.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:27.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:27.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:27.248+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:27.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:26 smithi026 ceph-mon[92215]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:27.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:26 smithi104 ceph-mon[81027]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:27.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:27.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:27.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:27.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:27.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:27.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:27.530+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:28.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:28.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:28.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:28.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:28.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:28.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:28.249+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:28.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:28.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:28.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:28.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:28.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:28.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:28.530+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:29.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:29.249+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:29.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:28 smithi026 ceph-mon[92215]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:28 smithi104 ceph-mon[81027]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:29.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:29.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:29.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:29.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:29.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:29.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:29.531+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:30.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:30.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:30.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:30.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:30.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:30.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:30.249+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:30.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:30.531+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:31.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:31.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:31.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:31.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:31.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:31.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:31.249+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:31.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:30 smithi026 ceph-mon[92215]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:30 smithi104 ceph-mon[81027]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:31.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.424+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004b00 con 0x55fc3f5009a0 2024-04-16T07:38:31.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.425+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.0 v2:172.21.15.26:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7fe9a0004b00 con 0x55fc3f5009a0 2024-04-16T07:38:31.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:31.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:31.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:31.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:31.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:31.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:31.531+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:32.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:32.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:32.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:32.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:32.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:32.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:32.249+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 56) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:32.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.531+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:32.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.532+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 25 ==== client_session(renewcaps seq 56) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:38:32.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:32.532+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 56) v1 from mds.0 2024-04-16T07:38:33.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:33.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:33.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:33.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:33.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:33.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:33.250+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:33.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:32 smithi026 ceph-mon[92215]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:33.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:32 smithi104 ceph-mon[81027]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:33.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:33.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:33.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:33.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:33.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:33.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:33.532+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:34.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:34.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:34.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:34.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:34.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:34.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:34.250+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:34.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:34.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:34.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:34.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:34.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:34.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:34.532+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:35.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:35.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:35.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:35.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:35.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:35.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:35.250+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:34 smithi026 ceph-mon[92215]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:35.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:34 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:35.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:34 smithi104 ceph-mon[81027]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:35.335 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:34 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:35.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:35.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:35.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:35.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:35.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:35.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:35.532+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:36.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:36.250+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:36.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:36.532+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:37.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:37.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:37.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:37.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:37.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:37.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:37.250+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:37.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:36 smithi026 ceph-mon[92215]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:37.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:36 smithi104 ceph-mon[81027]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:37.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:37.533+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:38.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:38.251+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:38.533+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:39.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:39.251+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:39.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:38 smithi026 ceph-mon[92215]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:39.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:38 smithi104 ceph-mon[81027]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:39.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:39.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:39.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:39.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:39.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:39.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:39.533+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:40.251+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:40.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:40.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:40.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:40.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:40.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:40.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:40.533+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:41.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.251+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:41.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.251+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:41.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.251+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:41.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.251+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:41.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.251+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:41.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:41.252+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:41.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:40 smithi026 ceph-mon[92215]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:41.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:40 smithi104 ceph-mon[81027]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:41.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:41.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:41.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:41.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:41.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:41.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:41.534+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:42.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:42.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:42.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:42.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:42.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:42.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:42.252+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:42.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:42.534+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:43.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:43.252+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:43.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:42 smithi026 ceph-mon[92215]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:43.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:42 smithi104 ceph-mon[81027]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:43.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:43.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:43.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:43.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:43.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:43.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:43.534+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:44.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:44.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:38:44.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 57) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.252+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:44.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.253+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 26 ==== client_session(renewcaps seq 57) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:38:44.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:44.253+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 57) v1 from mds.0 2024-04-16T07:38:44.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:44.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:44.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:44.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:44.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:44.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:44.534+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:45.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:45.253+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:45.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:44 smithi026 ceph-mon[92215]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:45.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:44 smithi104 ceph-mon[81027]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:45.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:45.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:45.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:45.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:45.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:45.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:45.535+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:46.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:46.253+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:46.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:46.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:46.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:46.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:46.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:46.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:46.535+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:47.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:47.253+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:47.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:46 smithi026 ceph-mon[92215]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:47.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:46 smithi104 ceph-mon[81027]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:47.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:47.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:47.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:47.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:47.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:47.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:47.535+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:48.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:48.253+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:48.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:48.536+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:49.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:49.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:49.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:49.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:49.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:49.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:49.254+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:49.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:48 smithi026 ceph-mon[92215]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:49.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:48 smithi104 ceph-mon[81027]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:49.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:49.536+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:50.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:50.254+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:50.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:49 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:50.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:49 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:38:50.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:50.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:50.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:50.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:50.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:50.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:50.536+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:51.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:51.254+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:51.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:50 smithi026 ceph-mon[92215]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:51.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:50 smithi104 ceph-mon[81027]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:51.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:51.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:51.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:51.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:51.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:51.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:51.536+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:52.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:52.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:52.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:52.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:52.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:52.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:52.255+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 57) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:52.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:52.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 26 ==== client_session(renewcaps seq 57) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:38:52.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:52.537+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 57) v1 from mds.0 2024-04-16T07:38:53.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:53.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:53.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:53.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:53.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:53.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:53.255+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:53.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:52 smithi026 ceph-mon[92215]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:53.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:52 smithi104 ceph-mon[81027]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:53.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:53.537+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:54.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:54.255+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:54.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:54.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:54.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:54.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:54.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:54.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:54.537+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:55.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:55.255+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:55.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:54 smithi026 ceph-mon[92215]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:55.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:54 smithi104 ceph-mon[81027]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:55.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:55.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:55.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:55.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:55.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:55.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:55.537+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:56.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.255+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:56.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.255+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:56.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.255+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:56.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.255+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:56.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.256+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:56.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:56.256+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:56.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:56.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:56.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:56.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:56.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:56.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:56.537+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:57.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:57.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:57.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:57.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:57.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:57.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:57.256+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:57.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:56 smithi026 ceph-mon[92215]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:57.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:56 smithi104 ceph-mon[81027]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:57.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:57.538+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:58.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:58.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:58.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:58.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:58.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:58.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:58.256+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:58.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:58.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:58.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:58.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:58.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:58.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:58.538+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:59.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:38:59.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:38:59.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:38:59.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:38:59.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:38:59.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:38:59.257+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:38:59.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:38:58 smithi026 ceph-mon[92215]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:59.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:38:58 smithi104 ceph-mon[81027]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:38:59.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:38:59.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:38:59.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:38:59.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:38:59.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:38:59.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:38:59.539+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:00.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:00.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:00.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:00.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:00.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:00.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:00.257+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:00.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:00.539+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:01.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:01.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:01.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:01.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:01.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:01.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:01.257+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:01.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:00 smithi026 ceph-mon[92215]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:01.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:00 smithi104 ceph-mon[81027]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:01.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:01.539+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:02.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:02.257+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:02.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:02.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:02.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:02.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:02.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:02.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:02.539+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:03.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:03.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:03.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:03.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:03.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:03.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:03.258+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:03.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:02 smithi026 ceph-mon[92215]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:03.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:02 smithi104 ceph-mon[81027]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:03.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:03.540+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 58) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:04.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.258+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:04.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.259+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 27 ==== client_session(renewcaps seq 58) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:39:04.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:04.259+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 58) v1 from mds.0 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:04.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:04.540+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:05.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:05.258+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:04 smithi026 ceph-mon[92215]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:05.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:04 smithi104 ceph-mon[81027]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:05.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:05.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:05.540+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:06.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:06.259+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:06.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:06.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:06.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:06.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:06.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:06.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:06.540+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:07.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.258+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:07.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.258+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:07.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.258+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:07.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.258+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:07.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.259+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:07.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:07.259+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:07.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:06 smithi026 ceph-mon[92215]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:07.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:06 smithi104 ceph-mon[81027]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:07.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:07.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:07.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:07.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:07.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:07.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:07.541+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:08.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:08.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:08.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:08.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:08.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:08.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:08.259+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:08.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:08.541+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:09.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:09.259+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:09.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:08 smithi026 ceph-mon[92215]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:09.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:09 smithi104 ceph-mon[81027]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:09.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:09.541+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:10.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:10.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:10.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:10.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:10.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:10.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:10.259+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:10.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:10.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:10.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:10.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:10.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:10.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:10.542+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:11.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:11.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:11.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:11.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:11.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:11.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:11.260+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:11.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:11 smithi026 ceph-mon[92215]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:11.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:11 smithi104 ceph-mon[81027]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:11.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:11.541+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:12.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:12.260+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 58) v5 -- 0x7fe9ac006a20 con 0x7fe9b4014240 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:12.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 27 ==== client_session(renewcaps seq 58) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:39:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:12.542+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 58) v1 from mds.0 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:13.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:13.261+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:13.261 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:13 smithi026 ceph-mon[92215]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:13.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:13 smithi104 ceph-mon[81027]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:13.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:13.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:13.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:13.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:13.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:13.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:13.542+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:14.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:14.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:14.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:14.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:14.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:14.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:14.260+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:14.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:14.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:14.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:14.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:14.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:14.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:14.543+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:15.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:15.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:15.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:15.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:15.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:15.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:15.261+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:15.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:15 smithi104 ceph-mon[81027]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:15.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:15 smithi026 ceph-mon[92215]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:15.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:15.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:15.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:15.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:15.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:15.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:15.542+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:16.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:16.261+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:16.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:16 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:39:16.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:16 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T07:39:16.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:16.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:16.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:16.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:16.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:16.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:16.541+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:17.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:17.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:17.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:17.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:17.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:17.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:17.262+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:17.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:17 smithi104 ceph-mon[81027]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:17.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:39:17.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:39:17.334 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:17 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:39:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:17 smithi026 ceph-mon[92215]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T07:39:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T07:39:17.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:17 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' 2024-04-16T07:39:17.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:17.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:17.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:17.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:17.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:17.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:17.542+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:18.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:18.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:18.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:18.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:18.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:18.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:18.262+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:18.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:18.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:18.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:18.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:18.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:18.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:18.543+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:19.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:19.262+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:19.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:19 smithi104 ceph-mon[81027]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:19.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:19 smithi026 ceph-mon[92215]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:19.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:19.543+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:20.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:20.263+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:20.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:20 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:20.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:20 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:20.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:20.543+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:21.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:21.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:21.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:21.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:21.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:21.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:21.262+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:21.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:21 smithi104 ceph-mon[81027]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:21.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:21 smithi026 ceph-mon[92215]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:21.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:21.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:21.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:21.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:21.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:21.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:21.543+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:22.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:22.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:22.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:22.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:22.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:22.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:22.263+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:22.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:22.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:22.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:22.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:22.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:22.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:22.544+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:23.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:23.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:23.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:23.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:23.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:23.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:23.263+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:23.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:23 smithi104 ceph-mon[81027]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:23.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:23 smithi026 ceph-mon[92215]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:23.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:23.544+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 59) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:24.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:24.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:24.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:24.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.263+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:24.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.264+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 28 ==== client_session(renewcaps seq 59) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:39:24.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:24.264+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 59) v1 from mds.0 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:24.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:24.544+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:25.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:25.263+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:25.336 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:25 smithi104 ceph-mon[81027]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:25.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:25 smithi026 ceph-mon[92215]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:25.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:25.544+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:26.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:26.264+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:26.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:26.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:26.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:26.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:26.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:26.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:26.545+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:27.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:27.264+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:27.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:27 smithi104 ceph-mon[81027]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:27.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:27 smithi026 ceph-mon[92215]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:27.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:27.545+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:28.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:28.265+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:28.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:28.545+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:29.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:29.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:29.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:29.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:29.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:29.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:29.265+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:29.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:29 smithi104 ceph-mon[81027]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:29.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:29 smithi026 ceph-mon[92215]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:29.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:29.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:29.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:29.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:29.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:29.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:29.546+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:30.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:30.265+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:30.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:30.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:30.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:30.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:30.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:30.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:30.546+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:31.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:31.265+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:31.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:31 smithi104 ceph-mon[81027]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:31.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:31 smithi026 ceph-mon[92215]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:31.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:31.546+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:32.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:32.266+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:32.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 59) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.546+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:32.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.547+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 28 ==== client_session(renewcaps seq 59) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9c8056040 con 0x7fe9b4014240 2024-04-16T07:39:32.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:32.547+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 59) v1 from mds.0 2024-04-16T07:39:33.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:33.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:33.266+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:33.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:33 smithi026 ceph-mon[92215]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:33.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:33.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:33.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:33.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:33.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:33.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:33.547+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:33.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:33 smithi104 ceph-mon[81027]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:34.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:34.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:34.266+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:34.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:34.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:34.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:34.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:34.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:34.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:34.547+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:35.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:35.266+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:35 smithi026 ceph-mon[92215]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:35 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:35.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:35.547+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:35.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:35 smithi104 ceph-mon[81027]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:35.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:35 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:36.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:36.267+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:36.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:36.547+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:37.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:37.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:37.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:37.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:37.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:37.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:37.267+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:37.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:37 smithi026 ceph-mon[92215]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:37.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:37.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:37.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:37.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:37.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:37.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:37.548+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:37.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:37 smithi104 ceph-mon[81027]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:38.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:38.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:38.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:38.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:38.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:38.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:38.267+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:38.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:38.548+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:39.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:39.267+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:39.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:39 smithi026 ceph-mon[92215]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:39.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:39.548+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:39.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:39 smithi104 ceph-mon[81027]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:40.267+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:40.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:40.548+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:41.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.267+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:41.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.267+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:41.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.267+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:41.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.267+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:41.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.268+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:41.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:41.268+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:41.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:41 smithi026 ceph-mon[92215]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:41.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:41.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:41.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:41.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:41.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:41.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:41.549+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:41.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:41 smithi104 ceph-mon[81027]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:42.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:42.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:42.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:42.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:42.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:42.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:42.268+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:42.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:42.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:42.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:42.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:42.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:42.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:42.549+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:43.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:43.268+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:43.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:43 smithi026 ceph-mon[92215]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:43.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:43.549+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:43.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:43 smithi104 ceph-mon[81027]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:44.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 60) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.268+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:44.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.269+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 29 ==== client_session(renewcaps seq 60) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:39:44.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:44.269+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 60) v1 from mds.0 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:44.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:44.549+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:45.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:45.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:45.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:45.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:45.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:45.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:45.269+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:45.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:45 smithi026 ceph-mon[92215]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:45.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:45.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:45.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:45.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:45.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:45.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:45.550+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:45.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:45 smithi104 ceph-mon[81027]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:46.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:46.269+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:46.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:46.550+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:47.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:47.269+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:47.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:47 smithi026 ceph-mon[92215]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:47.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:47.550+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:47.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:47 smithi104 ceph-mon[81027]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:48.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:48.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:48.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:48.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:48.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:48.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:48.270+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:48.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:48.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:48.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:48.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:48.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:48.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:48.551+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:49.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:49.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:49.270+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:49.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:49 smithi026 ceph-mon[92215]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:49.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:49.551+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:49.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:49 smithi104 ceph-mon[81027]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:50.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:50.270+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:50.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:50 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:50.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:50.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:50.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:50.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:50.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:50.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:50.551+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:50.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:50 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:39:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:51.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:51.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:51.270+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:51.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:51 smithi026 ceph-mon[92215]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:51.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:51.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:51.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:51.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:51.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:51.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:51.551+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:51.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:51 smithi104 ceph-mon[81027]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:52.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.270+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:52.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.271+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:52.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.271+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:52.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.271+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:52.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.271+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:52.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:52.271+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 60) v5 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:39:52.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006c30 con 0x7fe9b4014240 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.551+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.552+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 29 ==== client_session(renewcaps seq 60) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:52.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:52.552+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 60) v1 from mds.0 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:53.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:53.271+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:53.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:53 smithi026 ceph-mon[92215]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:53.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:53.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:53.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:53.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:53.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:53.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:53.552+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:53.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:53 smithi104 ceph-mon[81027]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:54.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:54.271+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:54.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:54.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:54.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:54.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:54.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:54.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:54.552+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:55.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:55.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:55.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:55.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:55.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:55.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:55.271+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:55.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:55 smithi026 ceph-mon[92215]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:55.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:55.552+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:55.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:55 smithi104 ceph-mon[81027]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:56.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:56.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:56.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:56.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:56.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:56.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:56.272+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:56.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:56.553+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:57.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:57.272+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:57.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:57 smithi026 ceph-mon[92215]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:57.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:57.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:57.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:57.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:57.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:57.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:57.553+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:57.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:57 smithi104 ceph-mon[81027]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:58.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:58.272+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:58.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:58.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:58.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:58.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:58.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:58.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:58.553+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:59.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:39:59.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:39:59.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:39:59.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:39:59.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:39:59.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:39:59.273+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:59.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:39:59 smithi026 ceph-mon[92215]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:39:59.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:39:59.553+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:39:59.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:39:59 smithi104 ceph-mon[81027]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:00.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:00.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:00.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:00.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:00.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:00.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:00.273+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:00.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:00 smithi026 ceph-mon[92215]: overall HEALTH_OK 2024-04-16T07:40:00.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:00.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:00.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:00.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:00.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:00.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:00.554+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:00.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:00 smithi104 ceph-mon[81027]: overall HEALTH_OK 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:01.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:01.273+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:01.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:01 smithi026 ceph-mon[92215]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:01.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:01.554+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:01.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:01 smithi104 ceph-mon[81027]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:02.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:02.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:02.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:02.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:02.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:02.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:02.273+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:02.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:02 smithi026 ceph-mon[92215]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:02.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:02.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:02.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:02.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:02.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:02.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:02.554+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:02.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:02 smithi104 ceph-mon[81027]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:03.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:03.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:03.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:03.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:03.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:03.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:03.274+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:03.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:03.554+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:04.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 61) v5 -- 0x7fbf68004560 con 0x7fbf7000cfc0 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:04.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:04.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mds.0 v2:172.21.15.26:6826/2688223128 30 ==== client_session(renewcaps seq 61) v1 ==== 28+0+0 (crc 0 0 0) 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:40:04.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:04.274+0000 7fbf77400640 10 client.14524 handle_client_session client_session(renewcaps seq 61) v1 from mds.0 2024-04-16T07:40:04.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:04.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:04.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:04.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:04.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:04.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:04.555+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:04 smithi026 ceph-mon[92215]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:05.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:04 smithi026 ceph-mon[92215]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:40:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:04 smithi104 ceph-mon[81027]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:05.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:04 smithi104 ceph-mon[81027]: from='mgr.34100 172.21.15.26:0/1511802318' entity='mgr.smithi026.tawexg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:05.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:05.274+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:05.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:05.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:05.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:05.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:05.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:05.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:05.555+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:06.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:06.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:06.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:06.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:06.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:06.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:06.275+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:06.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:06.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:06.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:06.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:06.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:06.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:06.555+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:07.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:06 smithi026 ceph-mon[92215]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:07.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:06 smithi104 ceph-mon[81027]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-16T07:40:07.208 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-16T07:40:07.209 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T07:40:07.211 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-16T07:40:07.223 INFO:tasks.cephadm:Teardown begin 2024-04-16T07:40:07.223 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/cephadm.py", line 2236, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T07:40:07.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:40:07.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.274+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:07.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.275+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:07.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.275+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:07.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.275+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:07.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.275+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:07.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:07.275+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:07.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:40:07.446 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-16T07:40:07.446 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T07:40:07.510 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T07:40:07.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:07.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:07.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:07.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:07.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:07.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:07.555+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:07.563 INFO:tasks.cephadm:Stopping all daemons... 2024-04-16T07:40:07.563 INFO:tasks.cephadm.mon.smithi026:Stopping mon.smithi026... 2024-04-16T07:40:07.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026 2024-04-16T07:40:08.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:07 smithi026 systemd[1]: Stopping Ceph mon.smithi026 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9b400d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-04-16T07:40:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9b400d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:40:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000a5c0 tx=0x7fe9d0005af0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:40:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000a5c0 tx=0x7fe9d0005af0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 reap_dead start 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9b400d1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.145+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400e3d0 con 0x7fe9b4011630 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.146+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400ecd0 con 0x7fe9b4016ff0 2024-04-16T07:40:08.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.146+0000 7fe9baa00640 0 client.24339 ms_handle_reset on v2:172.21.15.26:3300/0 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.147+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9b400e790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.147+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.147+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.149+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.145+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.145+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.145+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005930 tx=0x7fbf8c005af0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.145+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005930 tx=0x7fbf8c005af0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 reap_dead start 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf70008390 0x7fbf70010570 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000b120 con 0x7fbf7000b4e0 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000c770 con 0x7fbf7000c3a0 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.146+0000 7fbf77400640 0 client.14524 ms_handle_reset on v2:172.21.15.26:3300/0 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.147+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.147+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.147+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:08.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.148+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.152+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9b400e790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.152+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.152+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mdsmap=36+,monmap=4+}) v3 -- 0x7fe9a0004b00 con 0x7fe9b4016ff0 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.154+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9c001b4e0 tx=0x55fc3f51e640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.155+0000 7fe9baa00640 10 client.24339 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-16T07:40:08.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.150+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.150+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.150+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mdsmap=36+,monmap=4+}) v3 -- 0x7fbf5c004af0 con 0x7fbf7000c3a0 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.152+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbf80004d70 tx=0x7fbf80073000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.152+0000 7fbf77400640 10 client.14524 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.155+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf80053630 con 0x7fbf7000c3a0 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.155+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf80053790 con 0x7fbf7000c3a0 2024-04-16T07:40:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.157+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9ac003b50 con 0x7fe9b4016ff0 2024-04-16T07:40:08.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.167+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9c006d040 con 0x7fe9b4016ff0 2024-04-16T07:40:08.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:08.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:08.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:08.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:08.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:08.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:08.275+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:08.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:08 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[92211]: 2024-04-16T07:40:08.143+0000 7fbd57400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi026 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:08.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:08 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026[92211]: 2024-04-16T07:40:08.143+0000 7fbd57400700 -1 mon.smithi026@0(leader) e3 *** Got Signal Terminated *** 2024-04-16T07:40:08.512 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:08 smithi026 podman[147568]: 2024-04-16 07:40:08.308657646 +0000 UTC m=+0.321630426 container died 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:08.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:08.556+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:09.011 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:08 smithi026 podman[147568]: 2024-04-16 07:40:08.794248672 +0000 UTC m=+0.807221451 container cleanup 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-16T07:40:09.012 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:08 smithi026 bash[147568]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:09.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:09.275+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:09.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Apr 16 07:40:09 smithi026 podman[147593]: 2024-04-16 07:40:09.037308906 +0000 UTC m=+0.710036005 container remove 86cbf092af9f2e916dab5f9d0fa51e304e26ee3a4495e9fbcb7cc80e1d27ca2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi026, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux ) 2024-04-16T07:40:09.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:09.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:09.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:09.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:09.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:09.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:09.556+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:09.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi026.service' 2024-04-16T07:40:09.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:09.713 INFO:tasks.cephadm.mon.smithi026:Stopped mon.smithi026 2024-04-16T07:40:09.714 INFO:tasks.cephadm.mon.smithi104:Stopping mon.smithi104... 2024-04-16T07:40:09.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi104 2024-04-16T07:40:10.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:09 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:10.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:10.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.275+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:10.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 msgr2=0x7fe9b400a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-04-16T07:40:10.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 msgr2=0x7fe9b400a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:40:10.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9c001b4e0 tx=0x55fc3f51e640 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9c001b4e0 tx=0x55fc3f51e640 comp rx=0 tx=0).stop 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 msgr2=0x7fe9b400a990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9b4016ff0 0x7fe9b400a990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9b4011630 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 0 client.24339 ms_handle_reset on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9b400d1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9b400e790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.328+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 msgr2=0x7fbf7000fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 msgr2=0x7fbf7000fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbf80004d70 tx=0x7fbf80073000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbf80004d70 tx=0x7fbf80073000 comp rx=0 tx=0).stop 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 msgr2=0x7fbf7000fc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-16T07:40:10.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf7000c3a0 0x7fbf7000fc80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf70010510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.328+0000 7fbf77400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf7000b4e0 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf77400640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf77400640 0 client.14524 ms_handle_reset on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.329+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf70010510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.330+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.330+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9b400d1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9b400d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9b400e790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9b400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9b4011630 2024-04-16T07:40:10.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.440+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.441+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.530+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf70010510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:10.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:10.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:10.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:10.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:10.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:10.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.556+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:10.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:10 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104[81023]: 2024-04-16T07:40:10.326+0000 7f954de00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:10.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:10 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104[81023]: 2024-04-16T07:40:10.326+0000 7f954de00700 -1 mon.smithi104@1(peon) e3 *** Got Signal Terminated *** 2024-04-16T07:40:10.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:10 smithi104 podman[116999]: 2024-04-16 07:40:10.532108292 +0000 UTC m=+0.337682933 container died 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:10.642+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:10.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:10.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:10.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:10.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:10.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:10.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:10.931+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf70010510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:11.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:11.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:11.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:11.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:11.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:11.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.043+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:11 smithi104 podman[116999]: 2024-04-16 07:40:11.027530849 +0000 UTC m=+0.833105517 container cleanup 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T07:40:11.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:11 smithi104 bash[116999]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf70010510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf70010510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf70012d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf70012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:11.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf7000b4e0 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:11.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.154+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:11.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:11.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:11.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:11.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:11.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:11.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.276+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:11.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.355+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:11.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:11.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:11.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:11.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:11.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:11.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.556+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:11.583 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 16 07:40:11 smithi104 podman[117013]: 2024-04-16 07:40:11.245798095 +0000 UTC m=+0.692581384 container remove 08a1ef0af55e64c086738ba2bf9c6ff6a583bffb601317b3869f7b71d72b8334 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-mon-smithi104, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T07:40:11.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@mon.smithi104.service' 2024-04-16T07:40:11.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:11.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:11.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:11.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:11.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:11.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:11.756+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:11.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:11.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:11.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:11.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:11.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:11.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:11.844+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:11.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:11.883 INFO:tasks.cephadm.mon.smithi104:Stopped mon.smithi104 2024-04-16T07:40:11.883 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-16T07:40:11.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.0 2024-04-16T07:40:12.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:12 smithi026 systemd[1]: Stopping Ceph osd.0 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:12.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:12.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:12.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:12.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:12.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:12.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.276+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 61) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:12.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006c30 con 0x7fe9b4014240 2024-04-16T07:40:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9baa00640 1 -- 192.168.0.1:0/874387268 <== mds.0 v2:172.21.15.26:6826/2688223128 30 ==== client_session(renewcaps seq 61) v1 ==== 28+0+0 (crc 0 0 0) 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:40:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:12.557+0000 7fe9baa00640 10 client.24339 handle_client_session client_session(renewcaps seq 61) v1 from mds.0 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:12.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:12.557+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:12.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:40:12.496+0000 7fd6b6200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:12.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:40:12.496+0000 7fd6b6200700 -1 osd.0 85 *** Got signal Terminated *** 2024-04-16T07:40:12.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:12 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0[101189]: 2024-04-16T07:40:12.496+0000 7fd6b6200700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:13.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:13.276+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:13.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.446+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:13.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:13.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:13.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:13.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:13.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:13.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:13.556+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:14.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.159+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:14.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.159+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:14.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.159+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:14.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.160+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:14.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.160+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:14.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.160+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:14.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:14.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:14.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:14.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:14.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:14.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:14.277+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:14.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:14.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:14.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:14.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:14.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:14.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:14.557+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:15.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:15.277+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:15.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:15.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:15.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:15.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:15.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:15.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:15.557+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:16.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:16.277+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:16.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.558+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:16.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:16.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:16.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:16.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:16.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:16.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:16.650+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0002450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9b4011630 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9a0002990 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:17.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:17.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:17.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.191+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:17.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.192+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:17.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.278+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:17.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.363+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.363+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.363+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.363+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.364+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.364+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:17.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.392+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:17.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.392+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:17.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.392+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:17.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.393+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:17.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.393+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:17.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.393+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:17.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:17.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:17.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:17.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:17.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:17.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.558+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.793+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.793+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.793+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:17.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.794+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:17.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.794+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:17.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:17.794+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.904+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.904+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:17.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf7000b4e0 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:17.905+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:18.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.106+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:18.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.106+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:18.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.106+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:18.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.107+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:18.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.107+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:18.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.107+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:18.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:18.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:18.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:18.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:18.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:18.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.277+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:18.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:18 smithi026 podman[147701]: 2024-04-16 07:40:18.026768066 +0000 UTC m=+5.674990250 container died 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-16T07:40:18.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.507+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:18.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.507+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:18.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.507+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:18.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.508+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:18.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.508+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:18.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:18.508+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:18.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:18.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:18.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:18.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:18.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:18.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.558+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:18.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.594+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:18.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.594+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:18.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.594+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:18.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.595+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:18.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.596+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:18.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:18.596+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:18.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:18 smithi026 podman[147701]: 2024-04-16 07:40:18.422875703 +0000 UTC m=+6.071097914 container cleanup 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-16T07:40:18.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:18 smithi026 bash[147701]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0 2024-04-16T07:40:19.011 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:18 smithi026 podman[147722]: 2024-04-16 07:40:18.651895571 +0000 UTC m=+0.609904224 container remove 8c50566034f1dff9e4bb537c05f21bac9037eb042c42162d9983d59d33061bea (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-16T07:40:19.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:19.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:19.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:19.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:19.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:19.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.278+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.308+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.308+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.308+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.309+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.309+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:19.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:19.309+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:19.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:19.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:19.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:19.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:19.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:19.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:19.559+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:19.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:19 smithi026 podman[147807]: 2024-04-16 07:40:19.405731668 +0000 UTC m=+0.038962399 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:40:19.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:19 smithi026 podman[147807]: 2024-04-16 07:40:19.525171153 +0000 UTC m=+0.158401903 container create 31bf3feb2456495d0420fd4c4eb509edd6cb4bacd9c8a5e85db7f809dc185371 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-16T07:40:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.197+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.197+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:20.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.197+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:20.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.198+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:20.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.198+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:20.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.198+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:20.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:19 smithi026 podman[147807]: 2024-04-16 07:40:19.831219993 +0000 UTC m=+0.464450786 container init 31bf3feb2456495d0420fd4c4eb509edd6cb4bacd9c8a5e85db7f809dc185371 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T07:40:20.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:19 smithi026 podman[147807]: 2024-04-16 07:40:19.842901542 +0000 UTC m=+0.476132322 container start 31bf3feb2456495d0420fd4c4eb509edd6cb4bacd9c8a5e85db7f809dc185371 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:40:20.262 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:19 smithi026 podman[147807]: 2024-04-16 07:40:19.919099749 +0000 UTC m=+0.552330536 container attach 31bf3feb2456495d0420fd4c4eb509edd6cb4bacd9c8a5e85db7f809dc185371 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T07:40:20.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:20.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:20.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:20.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:20.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:20.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.278+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:20.558+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:20.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 16 07:40:20 smithi026 podman[147807]: 2024-04-16 07:40:20.310683364 +0000 UTC m=+0.943914168 container died 31bf3feb2456495d0420fd4c4eb509edd6cb4bacd9c8a5e85db7f809dc185371 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409) 2024-04-16T07:40:20.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.909+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:20.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.909+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:20.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.909+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:20.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.911+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:20.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.911+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:20.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:20.911+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:20.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-16T07:40:21.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:21.104 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-16T07:40:21.104 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-16T07:40:21.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.1 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:21.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:21.278+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:21.511 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:21 smithi026 systemd[1]: Stopping Ceph osd.1 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:21.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:21.559+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:22.011 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:21 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:40:21.710+0000 7f28b2e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:22.012 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:21 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:40:21.710+0000 7f28b2e00700 -1 osd.1 85 *** Got signal Terminated *** 2024-04-16T07:40:22.012 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:21 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1[106858]: 2024-04-16T07:40:21.710+0000 7f28b2e00700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:22.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:22.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:22.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:22.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:22.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:22.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:22.279+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:22.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:22.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:22.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:22.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:22.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:22.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:22.559+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:23.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:23.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:23.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:23.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:23.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:23.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:23.279+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:23.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.399+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:23.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.399+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:23.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.399+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.400+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.400+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:23.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.400+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:23.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:23.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:23.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:23.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:23.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:23.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:23.560+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:24.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.113+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.113+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:24.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.113+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:24.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.114+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:24.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.114+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:24.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.114+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 62) v5 -- 0x7fbf680047a0 con 0x7fbf7000cfc0 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:24.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:24.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:24.279+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:24.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:24.559+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:25.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:25.279+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:25.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:25.560+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:26.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:26.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:26.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:26.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:26.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:26.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:26.279+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:26.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:26.560+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.279+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.280+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.280+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.280+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.280+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:27.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:27.280+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 msgr2=0x7fe9a0002fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002990 0x7fe9a0002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9a0002450 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.317+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:27.511 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:27 smithi026 podman[147911]: 2024-04-16 07:40:27.249085088 +0000 UTC m=+5.680644540 container died 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T07:40:27.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:27.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:27.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:27.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:27.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:27.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.518+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:27.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.561+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:27.903 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:27 smithi026 podman[147911]: 2024-04-16 07:40:27.652461319 +0000 UTC m=+6.084020792 container cleanup 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:40:27.903 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:27 smithi026 bash[147911]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:27.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:27.919+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c002450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c002450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf7000b4e0 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.030+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf5c002990 2024-04-16T07:40:28.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:28.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.031+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:28.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.232+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:28.262 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:27 smithi026 podman[147931]: 2024-04-16 07:40:27.903387936 +0000 UTC m=+0.639256408 container remove 1988d8980b6fd0e9d06d74a3f977719d3cfedc0e9545fd97dcce9c652e0c027b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68004960 con 0x7fbf7000cfc0 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:28.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.280+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:28.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.561+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:28.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:28.633+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:28.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.720+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:28.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.720+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:28.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.720+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:28.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.720+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:28.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.721+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:28.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:28.721+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:29.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:28 smithi026 podman[148018]: 2024-04-16 07:40:28.681291073 +0000 UTC m=+0.037631601 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:40:29.075 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:28 smithi026 podman[148018]: 2024-04-16 07:40:28.812755917 +0000 UTC m=+0.169096465 container create 345efb67bbc04ed5e5638d96fc30f9e6f3a7980fa39866e6dee4553bbc168246 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5) 2024-04-16T07:40:29.075 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:29 smithi026 podman[148018]: 2024-04-16 07:40:29.074980995 +0000 UTC m=+0.431321586 container init 345efb67bbc04ed5e5638d96fc30f9e6f3a7980fa39866e6dee4553bbc168246 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:40:29.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:29.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:29.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:29.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:40:29.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:29.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.280+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:29.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:29.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:29.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:29.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:29.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:29.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:29.434+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:29.511 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:29 smithi026 podman[148018]: 2024-04-16 07:40:29.087266328 +0000 UTC m=+0.443606910 container start 345efb67bbc04ed5e5638d96fc30f9e6f3a7980fa39866e6dee4553bbc168246 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-16T07:40:29.511 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:29 smithi026 podman[148018]: 2024-04-16 07:40:29.181968182 +0000 UTC m=+0.538308704 container attach 345efb67bbc04ed5e5638d96fc30f9e6f3a7980fa39866e6dee4553bbc168246 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:29.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:29.561+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:29.884 INFO:journalctl@ceph.osd.1.smithi026.stdout:Apr 16 07:40:29 smithi026 podman[148044]: 2024-04-16 07:40:29.622236877 +0000 UTC m=+0.038553437 container died 345efb67bbc04ed5e5638d96fc30f9e6f3a7980fa39866e6dee4553bbc168246 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-16T07:40:30.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-16T07:40:30.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:30.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:30.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:30.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:40:30.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:30.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:30.281+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.322+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.322+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.322+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.323+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.323+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:30.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.323+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:30.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:30.336 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-16T07:40:30.336 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-16T07:40:30.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.2 2024-04-16T07:40:30.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:30.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:30.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:30.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:30.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:30.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:30.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:30.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:30 smithi026 systemd[1]: Stopping Ceph osd.2 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:31.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.036+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:31.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.036+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:31.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.036+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:31.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.037+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.037+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:31.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.037+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:31.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:31.281+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:31.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:31 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:40:31.019+0000 7f7f0f800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:31.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:31 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:40:31.019+0000 7f7f0f800700 -1 osd.2 85 *** Got signal Terminated *** 2024-04-16T07:40:31.512 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:31 smithi026 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2[112208]: 2024-04-16T07:40:31.019+0000 7f7f0f800700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:31.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:32.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:32.281+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:32.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:32.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:40:32.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:40:32.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:40:32.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 62) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:32.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:32.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:32.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:40:32.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:32.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:32.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:33.282+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:33.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.524+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:33.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.524+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:33.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.524+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:33.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.527+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:33.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.527+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:33.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.527+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:33.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:33.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:33.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:33.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:33.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:33.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:33.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:34.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.238+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:34.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.238+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:34.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.238+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:34.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.240+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:34.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.240+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:34.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.240+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:34.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:34.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:34.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:34.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:34.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:34.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:34.282+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:34.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:34.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:34.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:34.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:34.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:34.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:34.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:35.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:35.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:35.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:35.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:35.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:35.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:35.283+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:35.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:35.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:36.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:36.283+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:36.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:36.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:36.761 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:36 smithi026 podman[148123]: 2024-04-16 07:40:36.51190895 +0000 UTC m=+5.640254260 container died 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-16T07:40:37.064 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:36 smithi026 podman[148123]: 2024-04-16 07:40:36.862439585 +0000 UTC m=+5.990784956 container cleanup 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-16T07:40:37.064 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:36 smithi026 bash[148123]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2 2024-04-16T07:40:37.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:37.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:37.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:37.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:37.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:37.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:37.283+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:37.511 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:37 smithi026 podman[148154]: 2024-04-16 07:40:37.06324478 +0000 UTC m=+0.539155275 container remove 8f70feb33b9366b80c97524cfd2eaa354669b33a5672772ebc2c94a3e8c4d6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-16T07:40:37.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:37.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:37.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:37.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:37.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:37.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:37.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:38.151 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:37 smithi026 podman[148246]: 2024-04-16 07:40:37.80669164 +0000 UTC m=+0.035701066 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:40:38.151 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:37 smithi026 podman[148246]: 2024-04-16 07:40:37.906105578 +0000 UTC m=+0.135115021 container create 49fc4cad81d6158b9ac24fa743c8af3d4aad7b7742955ed024021b4d20a0a03b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T07:40:38.151 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:38 smithi026 podman[148246]: 2024-04-16 07:40:38.151514095 +0000 UTC m=+0.380523555 container init 49fc4cad81d6158b9ac24fa743c8af3d4aad7b7742955ed024021b4d20a0a03b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-16T07:40:38.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:38.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:38.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:38.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:38.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:38.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:38.283+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:38.514 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:38 smithi026 podman[148246]: 2024-04-16 07:40:38.163692163 +0000 UTC m=+0.392701598 container start 49fc4cad81d6158b9ac24fa743c8af3d4aad7b7742955ed024021b4d20a0a03b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:38.514 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:38 smithi026 podman[148246]: 2024-04-16 07:40:38.316828552 +0000 UTC m=+0.545838026 container attach 49fc4cad81d6158b9ac24fa743c8af3d4aad7b7742955ed024021b4d20a0a03b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:40:38.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:38.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:38.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:38.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:38.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:38.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:38.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:38.924 INFO:journalctl@ceph.osd.2.smithi026.stdout:Apr 16 07:40:38 smithi026 podman[148246]: 2024-04-16 07:40:38.634403613 +0000 UTC m=+0.863413100 container died 49fc4cad81d6158b9ac24fa743c8af3d4aad7b7742955ed024021b4d20a0a03b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T07:40:39.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:38 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:38.826+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:39.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-16T07:40:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:39.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:39.284+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:39.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:39.415 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-16T07:40:39.415 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-16T07:40:39.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.3 2024-04-16T07:40:39.555 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:39.298+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.562+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:39.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:39 smithi104 systemd[1]: Stopping Ceph osd.3 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:39.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.930+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:39.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.930+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:39.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.930+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:39.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.932+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:39.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.932+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:39.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:39.932+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:40.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:40.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:39.941+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:40.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:39 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:39.832+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:40.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:40.025+0000 7f73a9400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:40.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:40.025+0000 7f73a9400700 -1 osd.3 85 *** Got signal Terminated *** 2024-04-16T07:40:40.334 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:40.025+0000 7f73a9400700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:40.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:40.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:40.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:40.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:40.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:40.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:40.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:40.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.645+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:40.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.645+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:40.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.645+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:40.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.648+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:40.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.648+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:40.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:40.648+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:40.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:40.336+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:41.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:41.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:41.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:41.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:41.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:41.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:41.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:41.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:40.949+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:41.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:40 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:40.877+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:41.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:41.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:41.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:41.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:41.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:41.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:41.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:41.833 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:41 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:41.384+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:42.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:42.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:42.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:42.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:42.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:42.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:42.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:42.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:41 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:41.930+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:42.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:41 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:41.852+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:42.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:42.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:42.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 msgr2=0x7fe9a0003500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:42.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002450 0x7fe9a0003500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9a0002fc0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:42.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.504+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:42.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:42.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:42.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:42.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:42.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:42.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.563+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:42.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:42.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:42.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:42.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:42.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:42.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:42.705+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:42.828 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:42 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:42.390+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:43.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:42 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:42.964+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:43.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:42 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:42.828+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:43.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:43.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:43.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:43.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:43.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:43.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.106+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.218+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.218+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.218+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.218+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002990 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf5c002450 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:43.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:43.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:43.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:43.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:43.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.219+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:43.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:43.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.420+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:43.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:43.796 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:43 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:43.422+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:43.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:43.821+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:40:43.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:43.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:43.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:43.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:43.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:43.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:43.908+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:44.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:43 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:43.958+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:44.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:43 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:43.796+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:44.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:44.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:40:44.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:40:44.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 63) v5 -- 0x7fbf68004b20 con 0x7fbf7000cfc0 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:44.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:44.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:44.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:44.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:44.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:44.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:44.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:44.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:44.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.622+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:44.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.622+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:44.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.622+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:44.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.623+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:44.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.623+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:44.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:44.623+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:40:44.781 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:44 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3[90223]: 2024-04-16T07:40:44.402+0000 7f739fc00700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.100356+0000 front 2024-04-16T07:40:15.100169+0000 (oldest deadline 2024-04-16T07:40:39.199938+0000) 2024-04-16T07:40:45.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:44 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:44.780+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:45.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:44 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:44.909+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:45.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:45.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:45.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:45.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:45.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:45.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:45.285+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:45.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:45.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:45.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:45.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:45.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:45.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.509+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:45.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:45.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:45.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:45.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:45.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:45.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:45.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:45.781 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:45 smithi104 podman[117133]: 2024-04-16 07:40:45.491820044 +0000 UTC m=+5.618635349 container died d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-16T07:40:46.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:45 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:45.943+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:46.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:45 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:45.780+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:46.084 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:45 smithi104 podman[117133]: 2024-04-16 07:40:45.927499799 +0000 UTC m=+6.054315144 container cleanup d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T07:40:46.084 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:45 smithi104 bash[117133]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3 2024-04-16T07:40:46.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.224+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:46.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.224+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:46.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.224+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:46.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.225+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:46.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.225+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:40:46.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.225+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:46.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:46.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:46.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:46.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:46.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:46.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:46.286+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:46.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:46.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:46.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:46.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:46.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:46.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:46.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:46.583 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:46 smithi104 podman[117152]: 2024-04-16 07:40:46.178943287 +0000 UTC m=+0.673400500 container remove d9e22e0fb3cc31bc7d170e6448df3c6031dab857ccce19bc98f7eb3e6d91c3ec (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-16T07:40:47.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:46.972+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:47.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:46 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:46.796+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:47.084 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:46.917652004 +0000 UTC m=+0.031612410 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:40:47.084 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:47.038011722 +0000 UTC m=+0.151972157 container create 83f75327081d934b25ba57ba1d9ae5de0821021100feebd6d0e3957cdac027c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:47.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:47.286+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:47.397 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:47.299913197 +0000 UTC m=+0.413873692 container init 83f75327081d934b25ba57ba1d9ae5de0821021100feebd6d0e3957cdac027c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-16T07:40:47.397 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:47.312331105 +0000 UTC m=+0.426291516 container start 83f75327081d934b25ba57ba1d9ae5de0821021100feebd6d0e3957cdac027c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:47.398 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:47.39837921 +0000 UTC m=+0.512339648 container attach 83f75327081d934b25ba57ba1d9ae5de0821021100feebd6d0e3957cdac027c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:47.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:47.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:48.083 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 16 07:40:47 smithi104 podman[117247]: 2024-04-16 07:40:47.759374418 +0000 UTC m=+0.873334820 container died 83f75327081d934b25ba57ba1d9ae5de0821021100feebd6d0e3957cdac027c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:48.084 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:47 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:47.964+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:48.084 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:47 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:47.833+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:48.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:48.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:48.286+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:48.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:48.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.564+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:48.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:48.616 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-16T07:40:48.617 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-16T07:40:48.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.4 2024-04-16T07:40:48.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:48.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:48.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:48.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:48.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:48.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:48.712+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:49.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:48 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:48.868+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:49.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:48 smithi104 systemd[1]: Stopping Ceph osd.4 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:49.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:48 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:48.941+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:49.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:49.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:49.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:49.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:49.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:49.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.287+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.428+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.428+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.428+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.429+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.429+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:49.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:49.429+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:40:49.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.564+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:49.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:49.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.564+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:49.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.564+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:49.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.565+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:49.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:49.565+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:49.583 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:49.236+0000 7f87a5600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:49.583 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:49.236+0000 7f87a5600700 -1 osd.4 85 *** Got signal Terminated *** 2024-04-16T07:40:49.583 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:49.236+0000 7f87a5600700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:50.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:50.287+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:50.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:49.917+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:50.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:49.893+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:50.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:49 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:49.893+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:24.566746+0000 front 2024-04-16T07:40:24.566544+0000 (oldest deadline 2024-04-16T07:40:49.766578+0000) 2024-04-16T07:40:50.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:50.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:50.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:50.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:50.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:50.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:50.566+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:51.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:51.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:51.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:51.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:51.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:51.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:51.287+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:51.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:50 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:50.908+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:51.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:50 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:50.845+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:51.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:50 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:50.845+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:24.566746+0000 front 2024-04-16T07:40:24.566544+0000 (oldest deadline 2024-04-16T07:40:49.766578+0000) 2024-04-16T07:40:51.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:51.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:51.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:51.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:51.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:51.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:51.566+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:52.085 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:51 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:51.824+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:52.085 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:51 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:51.824+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:24.566746+0000 front 2024-04-16T07:40:24.566544+0000 (oldest deadline 2024-04-16T07:40:49.766578+0000) 2024-04-16T07:40:52.085 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:51 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:51.954+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:52.085 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:51 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:51.954+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.287+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.287+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.287+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.287+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.288+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:52.288+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:52.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:52.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:40:52.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 63) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:52.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:52.566+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:53.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:52 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:52.827+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:53.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:52 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:52.827+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:24.566746+0000 front 2024-04-16T07:40:24.566544+0000 (oldest deadline 2024-04-16T07:40:49.766578+0000) 2024-04-16T07:40:53.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:52 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:52.995+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:53.084 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:52 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:52.995+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:53.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:53.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:53.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:53.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:53.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:53.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:53.288+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:53.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.566+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:53.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:53.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.566+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:53.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.566+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:53.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.567+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:53.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:53.567+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:54.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:54.288+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:54.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:54 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:54.003+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:54.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:54 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:54.003+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:54.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:53 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:53.835+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:15.164960+0000 front 2024-04-16T07:40:15.165290+0000 (oldest deadline 2024-04-16T07:40:39.264747+0000) 2024-04-16T07:40:54.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:53 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4[95204]: 2024-04-16T07:40:53.835+0000 7f87a0e00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:24.566746+0000 front 2024-04-16T07:40:24.566544+0000 (oldest deadline 2024-04-16T07:40:49.766578+0000) 2024-04-16T07:40:54.566 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:54.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:54.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:54.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:54.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:54.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:54.567+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:55.083 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:54 smithi104 podman[117366]: 2024-04-16 07:40:54.784130482 +0000 UTC m=+5.709376554 container died 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:40:55.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:54 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:54.975+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:55.084 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:54 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:54.975+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:55.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.119+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:55.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:55.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:55.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:55.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:55.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:55.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.288+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:55.471 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:55 smithi104 podman[117366]: 2024-04-16 07:40:55.246345391 +0000 UTC m=+6.171591498 container cleanup 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:55.471 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:55 smithi104 bash[117366]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:55.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:55.567+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:55.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.832+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:40:55.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.832+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:40:55.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.832+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:55.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:55 smithi104 podman[117386]: 2024-04-16 07:40:55.472161429 +0000 UTC m=+0.672162437 container remove 37e9774e15c5305d67afb57006a42b14abba54fdda729ad82bd68e805cba98a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T07:40:55.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.835+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:40:55.835 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.835+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:40:55.835 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:55.835+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:40:56.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:56.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:56.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:56.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:56.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:56.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:56.289+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:56.289 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:55 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:55.962+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:56.289 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:55 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:55.962+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:56.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:56.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:56.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:56.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:56.568 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:56 smithi104 podman[117471]: 2024-04-16 07:40:56.19523205 +0000 UTC m=+0.036574319 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:40:56.568 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:56 smithi104 podman[117471]: 2024-04-16 07:40:56.306223589 +0000 UTC m=+0.147565895 container create af87fcc2a8d64e5cf92f1e94d2dae353fdf8237b47765af09ec6daa8ce575012 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:40:56.568 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:56 smithi104 podman[117471]: 2024-04-16 07:40:56.567545443 +0000 UTC m=+0.408887755 container init af87fcc2a8d64e5cf92f1e94d2dae353fdf8237b47765af09ec6daa8ce575012 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T07:40:56.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:56.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:56.567+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:56.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:56 smithi104 podman[117471]: 2024-04-16 07:40:56.579613112 +0000 UTC m=+0.420955425 container start af87fcc2a8d64e5cf92f1e94d2dae353fdf8237b47765af09ec6daa8ce575012 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T07:40:56.833 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:56 smithi104 podman[117471]: 2024-04-16 07:40:56.658496356 +0000 UTC m=+0.499838708 container attach af87fcc2a8d64e5cf92f1e94d2dae353fdf8237b47765af09ec6daa8ce575012 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True) 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:57.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:57.289+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:57.333 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 16 07:40:57 smithi104 podman[117471]: 2024-04-16 07:40:57.037990581 +0000 UTC m=+0.879332917 container died af87fcc2a8d64e5cf92f1e94d2dae353fdf8237b47765af09ec6daa8ce575012 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T07:40:57.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:56 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:56.919+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:57.334 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:56 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:56.919+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:57.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.567+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:57.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:57.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.567+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:57.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.567+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:57.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.568+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:57.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:57.568+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:57.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-16T07:40:57.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:40:57.876 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-16T07:40:57.876 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-16T07:40:57.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.5 2024-04-16T07:40:58.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:58.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:58.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:58.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:58.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:58.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:58.289+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:58.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:57 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:57.959+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:58.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:57 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:57.959+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:58.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:58 smithi104 systemd[1]: Stopping Ceph osd.5 for 4c4a53e6-fbc0-11ee-bc8f-c7b262605968... 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:58.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:58.568+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:58.833 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:58 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:58.528+0000 7fb59ce00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T07:40:58.833 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:58 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:58.528+0000 7fb59ce00700 -1 osd.5 85 *** Got signal Terminated *** 2024-04-16T07:40:58.833 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:58 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:58.528+0000 7fb59ce00700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T07:40:59.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:40:59.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:40:59.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:40:59.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:40:59.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:40:59.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:40:59.290+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:40:59.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:59 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:59.004+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:40:59.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:59 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:59.004+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:40:59.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:40:59.568+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:00.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:00.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:00.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:00.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:41:00.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:00.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:00.290+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:00.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:59 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:59.974+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:41:00.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:59 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:59.974+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:41:00.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:40:59 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:40:59.974+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-04-16T07:40:35.594905+0000 front 2024-04-16T07:40:35.595229+0000 (oldest deadline 2024-04-16T07:40:59.094678+0000) 2024-04-16T07:41:00.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:00.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:00.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:00.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:00.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:00.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:00.568+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:01.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:01.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:01.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:01.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:41:01.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:01.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:01.290+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:01.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:00 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:00.945+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:41:01.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:00 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:00.945+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:41:01.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:00 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:00.945+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-04-16T07:40:35.594905+0000 front 2024-04-16T07:40:35.595229+0000 (oldest deadline 2024-04-16T07:40:59.094678+0000) 2024-04-16T07:41:01.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:01.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:01.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:01.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:01.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:01.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:01.569+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:02.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:01 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:01.926+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:41:02.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:01 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:01.926+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:41:02.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:01 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:01.926+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-04-16T07:40:35.594905+0000 front 2024-04-16T07:40:35.595229+0000 (oldest deadline 2024-04-16T07:40:59.094678+0000) 2024-04-16T07:41:02.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:02.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:02.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:02.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:41:02.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:02.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:02.290+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:02.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:02.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:02.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:02.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:02.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:02.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:02.569+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:03.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:03.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:03.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:03.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003bb0 con 0x7fbf7000cfc0 2024-04-16T07:41:03.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:03.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:03.291+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:03.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:02 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:02.877+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-16T07:40:13.391476+0000 front 2024-04-16T07:40:13.391501+0000 (oldest deadline 2024-04-16T07:40:38.690971+0000) 2024-04-16T07:41:03.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:02 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:02.877+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-04-16T07:40:23.992762+0000 front 2024-04-16T07:40:23.992803+0000 (oldest deadline 2024-04-16T07:40:51.593027+0000) 2024-04-16T07:41:03.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:02 smithi104 ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5[100054]: 2024-04-16T07:41:02.877+0000 7fb598600700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-04-16T07:40:35.594905+0000 front 2024-04-16T07:40:35.595229+0000 (oldest deadline 2024-04-16T07:40:59.094678+0000) 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:03.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:03.569+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 10 client.14524 renew_caps() 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 15 client.14524 renew_caps requesting from mds.0 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 10 client.14524 renew_caps mds.0 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 64) v5 -- 0x7fbf68004ce0 con 0x7fbf7000cfc0 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:04.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:04.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:04.291+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:04.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:04 smithi104 podman[117577]: 2024-04-16 07:41:04.08002121 +0000 UTC m=+5.711004527 container died ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:41:04.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:04.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:04.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:04.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:04.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:04.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:04.569+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:04.664 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:04 smithi104 podman[117577]: 2024-04-16 07:41:04.488471523 +0000 UTC m=+6.119454880 container cleanup ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-16T07:41:04.664 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:04 smithi104 bash[117577]: ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5 2024-04-16T07:41:05.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:04 smithi104 podman[117600]: 2024-04-16 07:41:04.664282138 +0000 UTC m=+0.563074915 container remove ffa3ac0a534dbec14ea0a51e7713cec97d955d5ac47d4d5d6ead9ec11b001406 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9a0002fc0 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:05.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:05.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:05.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:05.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.286+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:05.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:05.292+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:05.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.487+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:05.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:05.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:05.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:05.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:05.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:05.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.570+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:05.793 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:05 smithi104 podman[117687]: 2024-04-16 07:41:05.408813072 +0000 UTC m=+0.034337826 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T07:41:05.793 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:05 smithi104 podman[117687]: 2024-04-16 07:41:05.523512942 +0000 UTC m=+0.149037720 container create ada394afadb651ca40073b7489e7306d6e0eb5adabcac6a8b8747caa7305deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T07:41:05.793 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:05 smithi104 podman[117687]: 2024-04-16 07:41:05.793602931 +0000 UTC m=+0.419127753 container init ada394afadb651ca40073b7489e7306d6e0eb5adabcac6a8b8747caa7305deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:05.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:05.888+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:06.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:06.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:06.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.000+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf5c002450 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:41:06.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:06.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:06.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:06.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:06.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:06.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.001+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:06.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:05 smithi104 podman[117687]: 2024-04-16 07:41:05.805701199 +0000 UTC m=+0.431225945 container start ada394afadb651ca40073b7489e7306d6e0eb5adabcac6a8b8747caa7305deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-16T07:41:06.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:05 smithi104 podman[117687]: 2024-04-16 07:41:05.950876421 +0000 UTC m=+0.576401188 container attach ada394afadb651ca40073b7489e7306d6e0eb5adabcac6a8b8747caa7305deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:06.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.202+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:06.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:06.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:06.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:06.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:06.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:06.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.292+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:06.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:06.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:06.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:06.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:06.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:06.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.570+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:06.583 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 16 07:41:06 smithi104 podman[117687]: 2024-04-16 07:41:06.264170342 +0000 UTC m=+0.889695106 container died ada394afadb651ca40073b7489e7306d6e0eb5adabcac6a8b8747caa7305deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T07:41:06.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:06.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:06.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:06.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:06.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:06.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:06.603+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:06.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:06.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:06.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:06.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:06.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:06.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:06.690+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:07.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c4a53e6-fbc0-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-16T07:41:07.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T07:41:07.129 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-16T07:41:07.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:07.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.292+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.404+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.404+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.404+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.405+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.405+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:07.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:07.405+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:07.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:07.570+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:08.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.292+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:08.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:08.292+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:08.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:08.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:08.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:08.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:08.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:08.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:08.571+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:09.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.006+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:09.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.292+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:09.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.292+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:09.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.292+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:09.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.292+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:09.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.293+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:09.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:09.293+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:09.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:09.571+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:10.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:10.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:10.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:10.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:10.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:10.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:10.293+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:10.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:10.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:10.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:10.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:10.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:10.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:10.571+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:11.293+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.495+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.496+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:11.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.496+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:11.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.496+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:11.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.496+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:11.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.496+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:11.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:11.571+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:12.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.209+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:12.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.209+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:12.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.209+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:12.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.210+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.210+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:12.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.210+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 20 client.14524 tick 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 20 client.14524 collect_and_send_metrics 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 20 client.14524 collect_and_send_global_metrics 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.003986, avg_latency: 0.001328, sq_sum: 1644727527978, count=3][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fbf68003c00 con 0x7fbf7000cfc0 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:12.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.293+0000 7fbf75600640 20 client.14524 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:12.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:12.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:41:12.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:41:12.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:41:12.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 64) v5 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006a70 con 0x7fe9b4014240 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:12.571+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stdout:ceph_fuse: fuse loop returned - joining 2024-04-16T07:41:12.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stdout:ceph_fuse: fuse loop returned - joined 2024-04-16T07:41:12.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 -1 fuse_session_loop_mt 3 -- done 2024-04-16T07:41:12.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:ceph-fuse[75906]: fuse finished with error 0 and tester_r 0 2024-04-16T07:41:12.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stdout:calling unmount 2024-04-16T07:41:12.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 2 client.14524 unmounting 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=4 ll_ref=350 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=4 ll_ref=350 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 _ll_drop_pins 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 _ll_put 0x7fbf70007b30 0x1 350 -> 0 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=5 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=5 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 _release 0x1.head(faked_ino=0 nref=6 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 _invalidate_inode_cache 0x1.head(faked_ino=0 nref=6 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 _flush 0x1.head(faked_ino=0 nref=7 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 nothing to flush 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=7 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_inode on 0x1.head(faked_ino=0 nref=7 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 1 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 flush_caps_sync 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 wait_sync_caps want 1 (last is 1, 0 total flushing) 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 _put_inode on 0x1.head(faked_ino=0 nref=7 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) n = 6 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 remove_cap mds.0 on 0x1.head(faked_ino=0 nref=1 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 15 client.14524 remove_cap last one, closing snaprealm 0x7fbf70007a40 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 20 client.14524 put_snap_realm 0x1 0x7fbf70007a40 1 -> 0 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 10 client.14524 _put_inode deleting 0x1.head(faked_ino=0 nref=1 ll_ref=0 cap_refs={1024=0} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=1 caps=- COMPLETE has_dir_layout 0x7fbf70007b30) 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 2 client.14524 _close_mds_session mds.0 seq 1 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_close seq 1) v5 -- 0x563cca8e5560 con 0x7fbf7000cfc0 2024-04-16T07:41:12.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:12.573+0000 7fbf94c4e300 2 client.14524 waiting for 1 mds session(s) to close (timeout: 30s) 2024-04-16T07:41:13.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.571+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:13.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:13.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.571+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:13.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.571+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:13.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.572+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:13.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:13.572+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:14.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:14.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:14.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:14.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:14.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:14.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:14.572+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:15.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:15.572+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:16.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:16.572+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:17.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:17.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:17.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:17.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:17.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:17.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.573+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:17.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.900+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:17.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.901+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:17.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.900+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:17.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.902+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:17.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.902+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:17.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:17.902+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:18.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:18.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:18.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:18.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:18.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:18.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:18.573+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:18.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.615+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:18.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.615+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:18.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.615+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:18.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.617+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:18.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.617+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:18.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:18.617+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:19.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:19.573+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:20.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:20.573+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:21.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:21.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:21.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:21.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:21.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:21.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:21.574+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:22.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:22.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:22.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:22.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:22.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:22.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:22.574+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:23.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:23.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:23.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:23.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:23.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:23.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:23.574+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:24.575+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:25.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:25.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:25.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:25.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:25.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:25.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:25.575+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:26.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:26.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:26.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:26.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:26.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:26.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:26.575+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:27.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-16T07:41:27.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T07:41:27.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:41:27.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-16T07:41:27.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.074+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.073+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 20 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.073+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.073+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.074+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.074+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.074+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.074+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:27.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.275+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.275+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.275+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:27.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.275+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.275+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.275+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:27.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:27.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:27.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:27.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac0062b0 con 0x7fe9b4014240 2024-04-16T07:41:27.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:27.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.575+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.676+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.676+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:27.676+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.676+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.676+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:27.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:27.676+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:28.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:28.477+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:28.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:28.477+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:28.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:28.477+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:28.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.477+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:28.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.478+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:28.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.478+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:28.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:28.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:28.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:28.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006c60 con 0x7fe9b4014240 2024-04-16T07:41:28.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:28.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:28.575+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:29.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:29.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:29.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:29.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac006fb0 con 0x7fe9b4014240 2024-04-16T07:41:29.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:29.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:29.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:30.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.080+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:30.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.080+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:30.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.080+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:30.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:30.079+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:30.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:30.079+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:30.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:30.079+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:30.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:30.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:30.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:30.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac007300 con 0x7fe9b4014240 2024-04-16T07:41:30.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:30.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:30.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.710+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:30.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.710+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:30.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.710+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:41:30.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.713+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:30.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.713+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:30.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:30.713+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:41:31.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.429+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:31.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.429+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:41:31.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.429+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:31.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.430+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:31.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.430+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:31.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:31.430+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:41:31.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:31.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:31.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:31.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac007650 con 0x7fe9b4014240 2024-04-16T07:41:31.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:31.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:31.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:32.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:32.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:41:32.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:41:32.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:41:32.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 65) v5 -- 0x7fe9ac0079a0 con 0x7fe9b4014240 2024-04-16T07:41:32.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:32.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:32.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac007b60 con 0x7fe9b4014240 2024-04-16T07:41:32.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:32.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:32.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:33.282+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:33.282+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:33.282+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.283+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.283+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:33.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.283+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:33.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac007eb0 con 0x7fe9b4014240 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:33.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:33.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac008200 con 0x7fe9b4014240 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:34.576+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:35.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f410 con 0x7fe9a0002fc0 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.286+0000 7fe9b9600640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b400f550 con 0x55fc3f5009a0 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:35.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:35.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:35.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.287+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:35.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:35.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:35.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:35.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:35.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:35.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.488+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:35.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.576+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:35.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:35.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.576+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:35.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.577+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac008580 con 0x7fe9b4014240 2024-04-16T07:41:35.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.577+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:35.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.577+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:35.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:35.889+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 msgr2=0x7fbf5c002fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c002450 0x7fbf5c002fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013600 con 0x7fbf5c003500 2024-04-16T07:41:36.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf76000640 1 -- 192.168.0.1:0/2108819864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70013740 con 0x7fbf70008390 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:36.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.002+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T07:41:36.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:36.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:36.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:36.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:36.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:36.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.203+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac008900 con 0x7fe9b4014240 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:36.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.577+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:36.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:36.604+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:36.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:36.690+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:37.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:37.406+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac008c80 con 0x7fe9b4014240 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:37.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:37.577+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:38.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:38.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.293+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac009000 con 0x7fe9b4014240 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:38.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:38.577+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:39.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:39.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:39.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:39.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:39.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:39.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.008+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac009380 con 0x7fe9b4014240 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:39.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.577+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:39.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.689+0000 7fbf91400640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:39.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.689+0000 7fbf91400640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:39.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:39.689+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:39.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.690+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:39.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.690+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:39.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:39.690+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:40.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:40.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:40.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:40.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac009700 con 0x7fe9b4014240 2024-04-16T07:41:40.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:40.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:40.578+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:41.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:41.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:41.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:41.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:41.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:41.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.497+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:41.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:41.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:41.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:41.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac009a80 con 0x7fe9b4014240 2024-04-16T07:41:41.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:41.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:41.578+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:42.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.211+0000 7fbf90a00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:42.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.211+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:42.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.211+0000 7fbf90a00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:42.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.212+0000 7fbf91e00640 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:42.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.212+0000 7fbf77400640 1 client.14524 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:42.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.212+0000 7fbf91e00640 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 client.14524 1 mds(s) did not respond to session close -- timing out. 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 5 client.14524 _closed_mds_session mds.0 seq 1 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 msgr2=0x7fbf70016e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=START_CONNECT pgs=9 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 remove_session_caps mds.0 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 kick_requests_closed for mds.0 2024-04-16T07:41:42.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 20 client.14524 put_snap_realm 0x3 0x7fbf7000bcb0 1 -> 0 2024-04-16T07:41:42.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 unmount stopping remount finisher 2024-04-16T07:41:42.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stdout:calling finalize 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 2 client.14524 unmounted. 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 client.14524 shutdown 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 shutdown stopping cache invalidator finisher 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 shutdown stopping dentry invalidator finisher 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 10 client.14524 shutdown stopping interrupt finisher 2024-04-16T07:41:42.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 msgr2=0x7fbf5c001f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 msgr2=0x7fbf5c003930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.574+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 shutdown_connections 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf70008390 0x7fbf5c001f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf5c003500 0x7fbf5c003930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 --2- 192.168.0.1:0/2108819864 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fbf7000cfc0 0x7fbf70016e10 unknown :-1 s=CLOSED pgs=9 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 >> 192.168.0.1:0/2108819864 conn(0x563cca969fe0 msgr2=0x563ccaa03ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 shutdown_connections 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.575+0000 7fbf94c4e300 1 -- 192.168.0.1:0/2108819864 wait complete. 2024-04-16T07:41:42.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi026.stderr:2024-04-16T07:41:42.576+0000 7fbf94c4e300 20 client.14524 trim_cache size 0 max 16384 2024-04-16T07:41:42.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:42.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:42.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:42.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac009e00 con 0x7fe9b4014240 2024-04-16T07:41:42.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:42.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:42.578+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00a180 con 0x7fe9b4014240 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:43.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:43.579+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:44.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:44.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:44.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:44.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00a500 con 0x7fe9b4014240 2024-04-16T07:41:44.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:44.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:44.579+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:45.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:45.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:45.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:45.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00a880 con 0x7fe9b4014240 2024-04-16T07:41:45.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:45.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:45.579+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00ac00 con 0x7fe9b4014240 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:46.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:46.579+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00af80 con 0x7fe9b4014240 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:47.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.579+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:47.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.899+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:41:47.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.899+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:47.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.899+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:41:47.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.903+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:41:47.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.904+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:41:47.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:47.904+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00b300 con 0x7fe9b4014240 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:48.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:48.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:49.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:49.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:49.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:49.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00b680 con 0x7fe9b4014240 2024-04-16T07:41:49.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:49.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:49.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:50.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:50.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:50.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:50.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00ba00 con 0x7fe9b4014240 2024-04-16T07:41:50.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:50.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:50.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:51.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:51.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:51.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:51.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00bd80 con 0x7fe9b4014240 2024-04-16T07:41:51.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:51.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:51.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:52.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.503+0000 7fe9d4a00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 msgr2=0x7fe9b4016620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:52.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.503+0000 7fe9d4a00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] conn(0x7fe9b4014240 0x7fe9b4016620 unknown :-1 s=START_CONNECT pgs=10 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:41:52.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.503+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:6826/2688223128 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 10 client.24339 renew_caps() 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 15 client.24339 renew_caps requesting from mds.0 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 10 client.24339 renew_caps mds.0 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_session(request_renewcaps seq 66) v5 -- 0x7fe9ac00c100 con 0x7fe9b4014240 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00c2c0 con 0x7fe9b4014240 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:52.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:52.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:52.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:41:52.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T07:41:52.925 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-16T07:41:52.926 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi026/crash 2024-04-16T07:41:52.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T07:41:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:tar: /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T07:41:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:tar: Error is not recoverable: exiting now 2024-04-16T07:41:53.049 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi104/crash 2024-04-16T07:41:53.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T07:41:53.166 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T07:41:53.166 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-04-16T07:41:53.170 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-16T07:41:53.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-16T07:41:53.288 INFO:tasks.cephadm:Compressing logs... 2024-04-16T07:41:53.288 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:41:53.292 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T07:41:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-16T07:41:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T07:41:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi026.log 2024-04-16T07:41:53.358 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi026.tawexg.log 2024-04-16T07:41:53.360 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi026.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T07:41:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi026.tawexg.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-16T07:41:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kqpyuh.log 2024-04-16T07:41:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-16T07:41:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kqpyuh.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T07:41:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T07:41:53.371 INFO:teuthology.orchestra.run.smithi026.stderr: 91.1% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T07:41:53.374 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log: /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log: 85.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T07:41:53.374 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-16T07:41:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T07:41:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kwjewe.log 2024-04-16T07:41:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kwjewe.log: 88.7% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T07:41:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-16T07:41:53.385 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T07:41:53.386 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.zwhiwr.log 2024-04-16T07:41:53.387 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.dpfdje.log 2024-04-16T07:41:53.388 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.zwhiwr.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T07:41:53.390 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.dpfdje.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-16T07:41:53.390 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T07:41:53.391 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-16T07:41:53.392 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-16T07:41:53.394 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T07:41:53.395 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T07:41:53.396 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi104.uqfljg.log 2024-04-16T07:41:53.396 INFO:teuthology.orchestra.run.smithi104.stderr: 91.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T07:41:53.397 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T07:41:53.398 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi104.uqfljg.log: 85.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T07:41:53.398 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi104.log 2024-04-16T07:41:53.402 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log: /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi104.log: gzip -5 --verbose -- /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T07:41:53.403 INFO:teuthology.orchestra.run.smithi104.stderr: 93.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.dpfdje.log.gz 2024-04-16T07:41:53.403 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T07:41:53.404 INFO:teuthology.orchestra.run.smithi104.stderr: 93.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi104.zwhiwr.log.gz 2024-04-16T07:41:53.425 INFO:teuthology.orchestra.run.smithi026.stderr: 90.9% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T07:41:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T07:41:53.449 INFO:teuthology.orchestra.run.smithi104.stderr: 89.8% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi104.uqfljg.log.gz 2024-04-16T07:41:53.457 INFO:teuthology.orchestra.run.smithi026.stderr: 93.7% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kqpyuh.log.gz 2024-04-16T07:41:53.486 INFO:teuthology.orchestra.run.smithi026.stderr: 94.1% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi026.kwjewe.log.gz 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00c670 con 0x7fe9b4014240 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:53.580+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:53.722 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mgr.smithi026.tawexg.log.gz 2024-04-16T07:41:53.726 INFO:teuthology.orchestra.run.smithi104.stderr: 92.6% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi104.log.gz 2024-04-16T07:41:53.994 INFO:teuthology.orchestra.run.smithi026.stderr: 93.2% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-16T07:41:54.003 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-16T07:41:54.009 INFO:teuthology.orchestra.run.smithi104.stderr: 93.7% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-16T07:41:54.126 INFO:teuthology.orchestra.run.smithi026.stderr: 93.4% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-16T07:41:54.128 INFO:teuthology.orchestra.run.smithi026.stderr: 93.5% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-16T07:41:54.157 INFO:teuthology.orchestra.run.smithi104.stderr: 93.6% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-16T07:41:54.163 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-16T07:41:54.163 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m0.816s 2024-04-16T07:41:54.163 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m2.248s 2024-04-16T07:41:54.163 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.297s 2024-04-16T07:41:54.416 INFO:teuthology.orchestra.run.smithi026.stderr: 90.3% -- replaced with /var/log/ceph/4c4a53e6-fbc0-11ee-bc8f-c7b262605968/ceph-mon.smithi026.log.gz 2024-04-16T07:41:54.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-16T07:41:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m1.095s 2024-04-16T07:41:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m3.562s 2024-04-16T07:41:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.355s 2024-04-16T07:41:54.420 INFO:tasks.cephadm:Archiving logs... 2024-04-16T07:41:54.421 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi026/log 2024-04-16T07:41:54.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:41:54.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:54.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:54.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:54.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00c9f0 con 0x7fe9b4014240 2024-04-16T07:41:54.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:54.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:54.581+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:55.047 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi104/log 2024-04-16T07:41:55.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T07:41:55.496 INFO:tasks.cephadm:Removing cluster... 2024-04-16T07:41:55.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 --force 2024-04-16T07:41:55.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:55.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:55.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:55.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00cd70 con 0x7fe9b4014240 2024-04-16T07:41:55.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:55.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:55.581+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:56.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c4a53e6-fbc0-11ee-bc8f-c7b262605968 --force 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00d0f0 con 0x7fe9b4014240 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:56.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:56.581+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:57.553 INFO:tasks.cephadm:Removing cephadm ... 2024-04-16T07:41:57.553 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00d470 con 0x7fe9b4014240 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:57.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:57.581+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:57.613 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T07:41:57.669 INFO:tasks.cephadm:Teardown complete 2024-04-16T07:41:57.669 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T07:41:57.681 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T07:41:57.682 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T07:41:57.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:41:57.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T07:41:57.831 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T07:41:57.842 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T07:41:57.842 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:41:57.851 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:41:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:41:57.906 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv01.front.sepia.ceph.com 3 7 377 88 +1611us[+1660us] +/- 49ms 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -483us[ -483us] +/- 66ms 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -2048us[-2048us] +/- 74ms 2024-04-16T07:41:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 7 377 25 +1608us[+1650us] +/- 50ms 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 23 -517us[ -517us] +/- 66ms 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -2068us[-2026us] +/- 74ms 2024-04-16T07:41:57.916 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:57.917 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T07:41:57.929 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T07:41:57.929 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T07:41:57.940 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:41:58.197 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T07:41:58.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:41:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:41:58.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.581+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.581+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.582+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00d7f0 con 0x7fe9b4014240 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.582+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:58.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:58.582+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:41:58.645 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T07:41:58.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T07:41:58.668 INFO:teuthology.task.internal:Duration was 2498.749323 seconds 2024-04-16T07:41:58.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T07:41:58.680 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T07:41:58.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:41:58.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T07:41:58.855 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:41:58.859 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:41:59.363 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T07:41:59.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T07:41:59.364 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:41:59.434 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-04-16T07:41:59.435 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T07:41:59.514 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T07:41:59.514 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:41:59.522 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T07:41:59.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:41:59.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:41:59.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:41:59.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00db70 con 0x7fe9b4014240 2024-04-16T07:41:59.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:41:59.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:41:59.582+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:41:59.623 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T07:41:59.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:41:59.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T07:42:00.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T07:42:00.425 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T07:42:00.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:42:00.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T07:42:00.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:42:00.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:42:00.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:42:00.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 3 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00def0 con 0x7fe9b4014240 2024-04-16T07:42:00.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:42:00.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.582+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:42:00.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T07:42:00.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:42:00.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:42:00.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.712+0000 7fe9cfe00640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 msgr2=0x7fe9a0003430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:3300/0 2024-04-16T07:42:00.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.712+0000 7fe9cfe00640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9a0002fc0 0x7fe9a0003430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:42:00.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.712+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.26:3300/0 2024-04-16T07:42:00.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.716+0000 7fe9d5400640 1 -- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 msgr2=0x7fe9a0001f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-04-16T07:42:00.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.716+0000 7fe9d5400640 1 --2- 192.168.0.1:0/874387268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fc3f5009a0 0x7fe9a0001f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-16T07:42:00.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:00.716+0000 7fe9baa00640 1 client.24339 ms_handle_refused on v2:172.21.15.104:3300/0 2024-04-16T07:42:00.763 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-04-16T07:42:00.765 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-04-16T07:42:00.833 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:42:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:42:00.884 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T07:42:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:42:00.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T07:42:00.950 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T07:42:00.951 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi026 2024-04-16T07:42:00.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:42:01.082 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944/remote/smithi104 2024-04-16T07:42:01.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T07:42:01.181 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T07:42:01.182 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T07:42:01.190 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T07:42:01.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T07:42:01.260 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T07:42:01.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T07:42:01.272 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T07:42:01.272 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:42:01.280 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T07:42:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: 393846 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 07:42 /home/ubuntu/cephtest 2024-04-16T07:42:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: 786708 4 d--------- 2 ubuntu ubuntu 4096 Apr 16 07:20 /home/ubuntu/cephtest/mnt.0 2024-04-16T07:42:01.332 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-16T07:42:01.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:unique: 702, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119149 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 8 client.24339 _ll_getattr 0x1.head 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 10 client.24339 _getattr mask pAsLsXsFs issued=0 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 choose_target_mds starting with req->inode 0x1.head(faked_ino=0 nref=5 ll_ref=351 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=2 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 choose_target_mds 0x1.head(faked_ino=0 nref=5 ll_ref=351 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=2 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) is_hash=0 hash=0 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 10 client.24339 choose_target_mds from caps on inode 0x1.head(faked_ino=0 nref=5 ll_ref=351 cap_refs={} open={} mode=41777 size=0/0 nlink=1 btime=2024-04-16T07:19:43.615094+0000 mtime=2024-04-16T07:19:43.615094+0000 ctime=2024-04-16T07:20:33.192545+0000 change_attr=2 caps=-(0=pAsLsXsFs) COMPLETE has_dir_layout 0x7fe9b4007b30) 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 mds is 0 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 10 client.24339 send_request rebuilding request 3 for mds.0 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 encode_cap_releases enter (req: 0x55fc3f500090, mds: 0) 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 send_request set sent_stamp to 2024-04-16T07:42:01.336538+0000 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 10 client.24339 send_request client_request(unknown.0:3 getattr pAsLsXsFs #0x1 2024-04-16T07:42:01.336491+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) v5 to mds.0 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_request(unknown.0:3 getattr pAsLsXsFs #0x1 2024-04-16T07:42:01.336491+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) v5 -- 0x55fc3f514360 con 0x7fe9b4014240 2024-04-16T07:42:01.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.335+0000 7fe9d8cc2300 20 client.24339 awaiting reply|forward|kick on 0x7fff23cd9300 2024-04-16T07:42:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:42:01.336 INFO:teuthology.orchestra.run.smithi026.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T07:42:01.337 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_a2afef7120fdaef83ddff164fdad44bc773a2630/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T07:42:01.337 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T07:42:01.357 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T07:42:01.370 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T07:42:01.371 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T07:42:01.394 DEBUG:teuthology.nuke:shortname: smithi026 2024-04-16T07:42:01.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:42:01.413 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 06:51:33.701669', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASqUji7Hg2IWkjy40yk/OzXBEAdIfVccvNsVVQ/E6HtRn4MKTqb+h1uO7U++GCAjvODh2qCgdlaUuqWd9LclDU='} 2024-04-16T07:42:01.417 DEBUG:teuthology.nuke:shortname: smithi104 2024-04-16T07:42:01.417 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:42:01.442 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_06:50:35-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 06:51:33.702665', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP742jjOnmsSmk4FOLUX8fmmQlWDuINa4UBsV3aBnYtJ2DZ+7N6h0yngVBpyYKn4C8bhJop0/cCuq344DfCVtG0='} 2024-04-16T07:42:01.474 INFO:teuthology.orchestra.console.smithi026:Power off 2024-04-16T07:42:01.474 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:42:01.501 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-04-16T07:42:01.501 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:01.518 INFO:teuthology.orchestra.console.smithi104:Power off 2024-04-16T07:42:01.519 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T07:42:01.528 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-16T07:42:01.540 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-04-16T07:42:01.540 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:01.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:42:01.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:42:01.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:42:01.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 4 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00e270 con 0x7fe9b4014240 2024-04-16T07:42:01.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:42:01.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:01.583+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:42:01.585 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-16T07:42:02.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:42:02.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:42:02.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:42:02.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 4 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00e5f0 con 0x7fe9b4014240 2024-04-16T07:42:02.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:42:02.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:02.583+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:42:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:42:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:42:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:42:03.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 4 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00e970 con 0x7fe9b4014240 2024-04-16T07:42:03.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:42:03.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:03.582+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:42:04.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 20 client.24339 tick 2024-04-16T07:42:04.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_metrics 2024-04-16T07:42:04.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 20 client.24339 collect_and_send_global_metrics 2024-04-16T07:42:04.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 1 -- 192.168.0.1:0/874387268 --> [v2:172.21.15.26:6826/2688223128,v1:172.21.15.26:6827/2688223128] -- client_metrics [client_metric_type: READ_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: WRITE_LATENCY latency: 0.000000, avg_latency: 0.000000, sq_sum: 0, count=0][client_metric_type: METADATA_LATENCY latency: 0.006789, avg_latency: 0.003394, sq_sum: 9766122710450, count=2][client_metric_type: CAP_INFO cap_hits: 348 cap_misses: 4 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 0 dlease_misses: 0 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 0 total_size: 0][client_metric_type: WRITE_IO_SIZES total_ops: 0 total_size: 0] v1 -- 0x7fe9ac00ecf0 con 0x7fe9b4014240 2024-04-16T07:42:04.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 20 client.24339 trim_cache size 0 max 16384 2024-04-16T07:42:04.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi104.stderr:2024-04-16T07:42:04.583+0000 7fe9b8c00640 20 client.24339 upkeep thread waiting interval 1.000000000s 2024-04-16T07:42:05.529 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:05.586 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:05.639 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-16T07:42:05.695 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-16T07:42:09.643 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:09.696 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T07:42:09.755 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-04-16T07:42:09.806 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-04-16T07:42:09.856 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-04-16T07:42:09.907 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-04-16T07:42:10.011 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2498.7493226528168 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 host.b-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 owner: scheduled_vshankar@teuthology status: fail success: false 2024-04-16T07:42:10.012 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:42:10.113 INFO:teuthology.run:FAIL